[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:10 dist-test-slave-t0tw krb5kdc[27035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:10 dist-test-slave-t0tw krb5kdc[27035](info): set up 2 sockets
Sep 28 16:43:10 dist-test-slave-t0tw krb5kdc[27035](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 16:43:11.361972 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:11 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
W20260928 16:43:11.715762 27028 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.26.101.62: real 0.195s user 0.000s sys 0.008s
I20260928 16:43:11.718525 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44567
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:12.244055 27048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:12.245263 27048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:12.245402 27048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:12.253451 27048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:12.253589 27048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:12.253651 27048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:12.253750 27048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:12.253819 27048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:12.265038 27048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:12.268573 27048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:12.271320 27048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:12.294127 27053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:12.294185 27054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:12.294159 27056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:12.294955 27048 server_base.cc:1034] running on GCE node
I20260928 16:43:12.297098 27048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:12.298772 27048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:12.300015 27048 hybrid_clock.cc:648] HybridClock initialized: now 1790613792299967 us; error 82 us; skew 500 ppm
I20260928 16:43:12.301232 27048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:12 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613792, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:12.305074 27048 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:12.306770 27048 webserver.cc:460] Webserver started at http://127.26.101.62:35297/ using document root <none> and password file <none>
I20260928 16:43:12.309048 27048 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:12.309204 27048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:12.309538 27048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:12.317621 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "86a37e1c00c6455c841b44316ee75d0c"
format_stamp: "Formatted at 2026-09-28 16:43:12 on dist-test-slave-t0tw"
server_key: "54c173262cffd288dd0a8e7b15040cdc"
server_key_iv: "4f4090566422004dd76291f5afe1c8fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:12.318478 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "86a37e1c00c6455c841b44316ee75d0c"
format_stamp: "Formatted at 2026-09-28 16:43:12 on dist-test-slave-t0tw"
server_key: "54c173262cffd288dd0a8e7b15040cdc"
server_key_iv: "4f4090566422004dd76291f5afe1c8fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:12.326190 27048 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260928 16:43:12.335281 27063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:12.336858 27048 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 16:43:12.337074 27048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "86a37e1c00c6455c841b44316ee75d0c"
format_stamp: "Formatted at 2026-09-28 16:43:12 on dist-test-slave-t0tw"
server_key: "54c173262cffd288dd0a8e7b15040cdc"
server_key_iv: "4f4090566422004dd76291f5afe1c8fd"
server_key_version: "encryptionkey@0"
I20260928 16:43:12.337288 27048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:12.356016 27048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:12.361167 27048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:12.361459 27048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:12.386333 27048 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:44567
I20260928 16:43:12.386339 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:44567 every 8 connection(s)
I20260928 16:43:12.389094 27048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:12.395032 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27048
I20260928 16:43:12.395401 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:12.398074 27028 external_mini_cluster.cc:1428] Setting key 7eeb590c06d5f8a2f720a4513f2e26f6
I20260928 16:43:12.398182 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:12.416826 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: Bootstrap starting.
I20260928 16:43:12.420562 27115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:12.422861 27115 log.cc:824] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:12.426141 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: No bootstrap required, opened a new log
Sep 28 16:43:12 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:12.439472 27115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } }
I20260928 16:43:12.440035 27115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:12.440169 27115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a37e1c00c6455c841b44316ee75d0c, State: Initialized, Role: FOLLOWER
I20260928 16:43:12.440377 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:12.405425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.26.101.62:44567)
0928 16:43:12.406512 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:12.407211 (+ 699us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:12.407819 (+ 608us) server_negotiation.cc:407] Connection header received
0928 16:43:12.417094 (+ 9275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:12.417120 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:12.417583 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:12.417990 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:12.421245 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:12.422615 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:12.424479 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:12.425192 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:12.425376 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:12.431406 (+ 6030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:12.431415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:12.431431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:12.431494 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:12.435037 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:12.435914 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:12.435922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:12.435928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:12.436110 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:12.436603 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:12.436606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:12.436609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:12.437212 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:12.437589 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:12.438478 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 16:43:12.438908 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":422,"threads_started":1}
I20260928 16:43:12.440910 27115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } }
I20260928 16:43:12.441169 27115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:12.441387 27115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:12.442615 27115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:12.462056 27115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } }
I20260928 16:43:12.464190 27115 leader_election.cc:302] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a37e1c00c6455c841b44316ee75d0c; no voters:
I20260928 16:43:12.465147 27115 leader_election.cc:288] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:12.465286 27120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:12.466173 27120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [term 1 LEADER]: Becoming Leader. State: Replica: 86a37e1c00c6455c841b44316ee75d0c, State: Running, Role: LEADER
I20260928 16:43:12.466776 27120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } }
I20260928 16:43:12.470302 27115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:43:12.484449 27135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:12.484658 27135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:12.485965 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a37e1c00c6455c841b44316ee75d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } } }
I20260928 16:43:12.486431 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:12.485901 27122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a37e1c00c6455c841b44316ee75d0c. Latest consensus state: current_term: 1 leader_uuid: "86a37e1c00c6455c841b44316ee75d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a37e1c00c6455c841b44316ee75d0c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44567 } } }
I20260928 16:43:12.486570 27122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:12.488008 27137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:12.490177 27137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:12.500138 27137 catalog_manager.cc:1347] Generated new cluster ID: 937fe1d216e94f7ca80ca1125d2869b8
I20260928 16:43:12.500288 27137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:12.514247 27137 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:12.515734 27137 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:12.524188 27137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a37e1c00c6455c841b44316ee75d0c: Generated new TSK 0
I20260928 16:43:12.525313 27137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
W20260928 16:43:13.275478 27028 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.26.101.1: real 0.735s user 0.001s sys 0.009s
I20260928 16:43:13.276521 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:13.640215 27141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:13.640601 27141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:13.640771 27141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:13.649052 27141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:13.649219 27141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:13.649428 27141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:13.660203 27141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:13.661644 27141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:13.663251 27141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:13.678167 27146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:13.678192 27147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:13.678428 27149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:13.678946 27141 server_base.cc:1034] running on GCE node
I20260928 16:43:13.679611 27141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:13.680612 27141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:13.681927 27141 hybrid_clock.cc:648] HybridClock initialized: now 1790613793681866 us; error 64 us; skew 500 ppm
I20260928 16:43:13.682318 27141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:13 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613793, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:13.685897 27141 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:13.687553 27141 webserver.cc:460] Webserver started at http://127.26.101.1:46607/ using document root <none> and password file <none>
I20260928 16:43:13.688270 27141 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:13.688395 27141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:13.688706 27141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:13.691828 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "40b2308139ef442d930b4e37255adf88"
format_stamp: "Formatted at 2026-09-28 16:43:13 on dist-test-slave-t0tw"
server_key: "6ea081e692074f9306942f304a2109a1"
server_key_iv: "f34bdc47a99e8b1abbd12e96ef787ceb"
server_key_version: "encryptionkey@0"
I20260928 16:43:13.692629 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "40b2308139ef442d930b4e37255adf88"
format_stamp: "Formatted at 2026-09-28 16:43:13 on dist-test-slave-t0tw"
server_key: "6ea081e692074f9306942f304a2109a1"
server_key_iv: "f34bdc47a99e8b1abbd12e96ef787ceb"
server_key_version: "encryptionkey@0"
I20260928 16:43:13.698268 27141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:13.702399 27156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:13.704036 27141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:13.704195 27141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "40b2308139ef442d930b4e37255adf88"
format_stamp: "Formatted at 2026-09-28 16:43:13 on dist-test-slave-t0tw"
server_key: "6ea081e692074f9306942f304a2109a1"
server_key_iv: "f34bdc47a99e8b1abbd12e96ef787ceb"
server_key_version: "encryptionkey@0"
I20260928 16:43:13.704404 27141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:13.726025 27141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:13.730821 27141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:13.731125 27141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:13.732182 27141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:13.733734 27141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:13.733817 27141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:13.733930 27141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:13.733976 27141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:13.769403 27141 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:36735
I20260928 16:43:13.769492 27268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:36735 every 8 connection(s)
I20260928 16:43:13.770897 27141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:13.774175 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27141
I20260928 16:43:13.774420 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:13.775213 27028 external_mini_cluster.cc:1428] Setting key 448aabccb82d65b92cbe051a600b238b
Sep 28 16:43:13 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613793, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:43:13.796546 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:13.776263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57337 (local address 127.26.101.62:44567)
0928 16:43:13.776748 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:13.776753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:13.778281 (+ 1528us) server_negotiation.cc:407] Connection header received
0928 16:43:13.781133 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:13.781138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:13.781210 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:13.781310 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:13.783482 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:13.784514 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:13.786301 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:13.786703 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:13.786821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:13.790488 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:13.790493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:13.790497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:13.790532 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:13.793144 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:13.793944 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:13.793949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:13.793951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:13.794096 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:13.794542 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:13.794547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:13.794551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:13.794854 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:13.795002 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:13.795838 (+ 836us) server_negotiation.cc:299] Negotiation successful
0928 16:43:13.796150 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":268,"threads_started":1}
I20260928 16:43:13.797295 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:13.777013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44567 (local address 127.26.101.1:57337)
0928 16:43:13.778087 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:13.778145 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:13.780755 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:13.781595 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:13.781608 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:13.782416 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:13.783259 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:13.783283 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:13.784704 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:13.784708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:13.786131 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:13.786147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:13.786952 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:13.786960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:13.787239 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:13.787923 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:13.787954 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:13.790234 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:13.793298 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:13.793308 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:13.793328 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:13.793787 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:13.794205 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:13.794211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:13.794215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:13.794422 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:13.795004 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:13.795013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:13.795442 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 16:43:13.795692 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 16:43:13.796133 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":410,"threads_started":1}
I20260928 16:43:13.799094 27269 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44567
I20260928 16:43:13.799438 27269 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:13.800249 27269 heartbeater.cc:511] Master 127.26.101.62:44567 requested a full tablet report, sending...
I20260928 16:43:13.803946 27080 ts_manager.cc:194] Registered new tserver with Master: 40b2308139ef442d930b4e37255adf88 (127.26.101.1:36735)
I20260928 16:43:13.806604 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:57337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:13.818922 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:14.185186 27275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:14.185562 27275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:14.185765 27275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:14.194085 27275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:14.194316 27275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:14.194511 27275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:14.205090 27275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:14.206459 27275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:14.207928 27275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:14.221676 27280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:14.221621 27283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:14.221585 27281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:14.222532 27275 server_base.cc:1034] running on GCE node
I20260928 16:43:14.223292 27275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:14.224260 27275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:14.225498 27275 hybrid_clock.cc:648] HybridClock initialized: now 1790613794225411 us; error 45 us; skew 500 ppm
I20260928 16:43:14.225857 27275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:14.229363 27275 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:14.230911 27275 webserver.cc:460] Webserver started at http://127.26.101.2:32959/ using document root <none> and password file <none>
I20260928 16:43:14.231647 27275 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:14.231752 27275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:14.232062 27275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:14.235215 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "37190478047647cfa886bada4eb58249"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "828a9123e9b3e07a76aa0d00cbb90a74"
server_key_iv: "f2e27d261b5dc94cda81cfdda95c38fb"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.236044 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "37190478047647cfa886bada4eb58249"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "828a9123e9b3e07a76aa0d00cbb90a74"
server_key_iv: "f2e27d261b5dc94cda81cfdda95c38fb"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.241374 27275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:14.245193 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.246559 27275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:14.246704 27275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "37190478047647cfa886bada4eb58249"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "828a9123e9b3e07a76aa0d00cbb90a74"
server_key_iv: "f2e27d261b5dc94cda81cfdda95c38fb"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.246896 27275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:14.262318 27275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:14.266837 27275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:14.267182 27275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:14.268441 27275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:14.269944 27275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:14.270021 27275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.270138 27275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:14.270187 27275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.312457 27275 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:45995
I20260928 16:43:14.312490 27402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:45995 every 8 connection(s)
I20260928 16:43:14.313985 27275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:14.317150 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27275
I20260928 16:43:14.317363 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:14.317938 27028 external_mini_cluster.cc:1428] Setting key a8a0bb09c399ca505c80272ae193205e
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:43:14.336156 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.316724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:46305 (local address 127.26.101.62:44567)
0928 16:43:14.317197 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:14.317202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:14.318373 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 16:43:14.320545 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:14.320551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:14.320633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:14.320724 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:14.322501 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.323356 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.324984 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.325412 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.325525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.329942 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:14.329948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:14.329951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:14.329987 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:14.333070 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.333800 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.333806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.333810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.333978 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.334387 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.334391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.334392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.334710 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:14.334830 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:14.335633 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 16:43:14.335863 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260928 16:43:14.337206 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.317324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44567 (local address 127.26.101.2:46305)
0928 16:43:14.318197 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:14.318244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:14.320270 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:14.321038 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:14.321048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:14.321620 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:14.322321 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:14.322344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.323517 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.323521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:14.324815 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:14.324828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.325657 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.325666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:14.325966 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.326713 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:14.326737 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:14.329672 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:14.333214 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:14.333222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:14.333237 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:14.333681 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:14.334070 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:14.334073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:14.334076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:14.334299 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:14.334808 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:14.334813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:14.335194 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 16:43:14.335443 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 16:43:14.336018 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":342,"threads_started":1}
I20260928 16:43:14.338689 27403 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44567
I20260928 16:43:14.338994 27403 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:14.339738 27403 heartbeater.cc:511] Master 127.26.101.62:44567 requested a full tablet report, sending...
I20260928 16:43:14.341221 27080 ts_manager.cc:194] Registered new tserver with Master: 37190478047647cfa886bada4eb58249 (127.26.101.2:45995)
I20260928 16:43:14.342348 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:46305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:14.358258 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:14.733119 27409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:14.733444 27409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:14.733587 27409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:14.741483 27409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:14.741698 27409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:14.741873 27409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:14.752846 27409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:14.754209 27409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:14.755563 27409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:14.769567 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:14.770488 27414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:14.771248 27409 server_base.cc:1034] running on GCE node
W20260928 16:43:14.774770 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:14.775547 27409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:14.776569 27409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:14.777781 27409 hybrid_clock.cc:648] HybridClock initialized: now 1790613794777735 us; error 34 us; skew 500 ppm
I20260928 16:43:14.778120 27409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:14.785660 27409 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:14.787715 27409 webserver.cc:460] Webserver started at http://127.26.101.3:34813/ using document root <none> and password file <none>
I20260928 16:43:14.788569 27409 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:14.788707 27409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:14.789037 27409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:14.792456 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "bd70f6b991af46f8b3e2146c57e28597"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "447525edb9627b7bd57ccbb153970fd7"
server_key_iv: "0fd026ce51097467f01b0867af67c4ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.793288 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "bd70f6b991af46f8b3e2146c57e28597"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "447525edb9627b7bd57ccbb153970fd7"
server_key_iv: "0fd026ce51097467f01b0867af67c4ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.799540 27409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 16:43:14.803655 27424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.805249 27409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:43:14.805429 27409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "bd70f6b991af46f8b3e2146c57e28597"
format_stamp: "Formatted at 2026-09-28 16:43:14 on dist-test-slave-t0tw"
server_key: "447525edb9627b7bd57ccbb153970fd7"
server_key_iv: "0fd026ce51097467f01b0867af67c4ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:14.805642 27409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:14.810463 27269 heartbeater.cc:503] Master 127.26.101.62:44567 was elected leader, sending a full tablet report...
I20260928 16:43:14.822163 27409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:14.826426 27409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:14.826726 27409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:14.827876 27409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:14.829406 27409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:14.829479 27409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.829572 27409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:14.829627 27409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:14.870766 27409 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:44661
I20260928 16:43:14.870882 27536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:44661 every 8 connection(s)
I20260928 16:43:14.872243 27409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:14.876475 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27409
I20260928 16:43:14.876669 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:14.877272 27028 external_mini_cluster.cc:1428] Setting key 6e5f0fc793485151ff56e19b79bd25fd
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:14.895234 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.874860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:40765 (local address 127.26.101.62:44567)
0928 16:43:14.875415 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:14.875419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:14.876539 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 16:43:14.878873 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:14.878880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:14.878969 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:14.879121 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:14.881033 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.881847 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.883872 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.884325 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.884448 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.888482 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:14.888487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:14.888491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:14.888527 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:14.891525 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.892553 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.892558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.892561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.892743 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.893325 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.893330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.893333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.893760 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:14.893964 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:14.894685 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 16:43:14.894864 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":313,"threads_started":1}
I20260928 16:43:14.896124 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.875605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44567 (local address 127.26.101.3:40765)
0928 16:43:14.876386 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:14.876424 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:14.878553 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:14.879448 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:14.879461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:14.880215 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:14.880823 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:14.880844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.882066 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.882070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:14.883723 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:14.883736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.884541 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.884546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:14.884760 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.885442 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:14.885466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:14.888192 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:14.891712 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:14.891722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:14.891740 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:14.892397 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:14.892863 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:14.892868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:14.892872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:14.893202 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:14.893907 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:14.893915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:14.894334 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 16:43:14.894575 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 16:43:14.895003 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":305,"threads_started":1}
I20260928 16:43:14.897437 27537 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44567
I20260928 16:43:14.897719 27537 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:14.898665 27537 heartbeater.cc:511] Master 127.26.101.62:44567 requested a full tablet report, sending...
I20260928 16:43:14.900198 27080 ts_manager.cc:194] Registered new tserver with Master: bd70f6b991af46f8b3e2146c57e28597 (127.26.101.3:44661)
I20260928 16:43:14.901054 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:40765
I20260928 16:43:14.906750 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:14.954694 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.936761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.26.101.1:36735)
0928 16:43:14.937470 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:14.937477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:14.937505 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:43:14.937570 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:14.937577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:14.937795 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:14.937941 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:14.939894 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.940949 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.942475 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.943176 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.943281 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.947673 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:14.947689 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:14.947707 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:14.947750 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:14.951266 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.952148 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.952157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.952159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.952373 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:14.952933 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:14.952954 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:14.952958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:14.953456 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:14.953728 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:14.954064 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 16:43:14.954306 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":386,"threads_started":1}
W20260928 16:43:14.956130 27184 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59364
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:43:14 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:15.003687 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:14.987229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.26.101.1:36735)
0928 16:43:14.987489 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:14.987495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:14.987725 (+ 230us) server_negotiation.cc:407] Connection header received
0928 16:43:14.987913 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:14.987917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:14.988023 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:14.988115 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:14.989966 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.990804 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.992317 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:14.992795 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:14.992910 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:14.997505 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:14.997512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:14.997517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:14.997565 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:15.000502 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.001383 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.001387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.001390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.001560 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.002096 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.002100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.002102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.002423 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:15.002624 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:15.003147 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 16:43:15.003361 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 16:43:15.005687 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27141
I20260928 16:43:15.021593 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:15.007031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.26.101.1:36735)
0928 16:43:15.007290 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:15.007297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:15.007656 (+ 359us) server_negotiation.cc:407] Connection header received
0928 16:43:15.007822 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:15.007827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:15.007977 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:15.008096 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:15.009619 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.010578 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.012102 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.012530 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.012648 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:15.015411 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:15.015417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:15.015420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:15.015461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:15.018553 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.019458 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.019462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.019465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.019618 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.020074 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.020079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.020081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.020386 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:15.020637 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:15.020991 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 16:43:15.021221 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 16:43:15.022380 27264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:15.108242 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27141
I20260928 16:43:15.124655 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27275
Sep 28 16:43:15 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:15.142444 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:15.125603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.26.101.2:45995)
0928 16:43:15.126195 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:15.126202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:15.126230 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:43:15.126292 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:15.126298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:15.126529 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:15.126668 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:15.128262 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.129307 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.130712 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.131378 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.131492 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:15.135760 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:15.135776 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:15.135793 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:15.135837 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:15.138809 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.139649 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.139658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.139660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.139814 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.140312 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.140316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.140319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.140729 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:15.140992 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:15.141778 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 16:43:15.142011 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":328,"threads_started":1}
I20260928 16:43:15.143958 27398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:15.226549 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27275
I20260928 16:43:15.246052 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27409
Sep 28 16:43:15 dist-test-slave-t0tw krb5kdc[27035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:15.262912 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:15.247144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.26.101.3:44661)
0928 16:43:15.247789 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:15.247796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:15.247826 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:43:15.247885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:15.247892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:15.248120 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:15.248277 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:15.249561 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.250580 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.251898 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:15.252490 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:15.252606 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:15.256722 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:15.256733 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:15.256750 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:15.256786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:15.259947 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.260697 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.260705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.260707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.260864 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:15.261237 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:15.261241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:15.261243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:15.261629 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:15.261956 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:15.262292 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 16:43:15.262567 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":306,"threads_started":1}
I20260928 16:43:15.264340 27532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:15.347776 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27409
I20260928 16:43:15.363691 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27048
I20260928 16:43:15.364966 27110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:15.470098 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27048
2026-09-28T16:43:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5160 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:15 dist-test-slave-t0tw krb5kdc[27559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:15 dist-test-slave-t0tw krb5kdc[27559](info): set up 2 sockets
Sep 28 16:43:15 dist-test-slave-t0tw krb5kdc[27559](info): commencing operation
krb5kdc: starting...
W20260928 16:43:16.470449 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:16 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:16.646368 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45605
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:16.962975 27572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:16.963333 27572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:16.963470 27572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:16.971992 27572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:16.972108 27572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:16.972158 27572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:16.972239 27572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:16.972297 27572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:16.982354 27572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:16.983765 27572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:16.985162 27572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:16.996728 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:16.996702 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:16.996763 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:16.997649 27572 server_base.cc:1034] running on GCE node
I20260928 16:43:16.998472 27572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:17.000047 27572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:17.001286 27572 hybrid_clock.cc:648] HybridClock initialized: now 1790613797001233 us; error 44 us; skew 500 ppm
I20260928 16:43:17.001659 27572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:17 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613797, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:17.004911 27572 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:17.006469 27572 webserver.cc:460] Webserver started at http://127.26.101.62:40255/ using document root <none> and password file <none>
I20260928 16:43:17.007241 27572 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:17.007380 27572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:17.007700 27572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:17.010950 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "c1f9900226704f7a956d88bd72fa262b"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "11e0bac2d0c59c6515304a83fabc8fdc"
server_key_iv: "1894ca53036bbbb787891ec256303b6e"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.011778 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "c1f9900226704f7a956d88bd72fa262b"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "11e0bac2d0c59c6515304a83fabc8fdc"
server_key_iv: "1894ca53036bbbb787891ec256303b6e"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.017390 27572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:17.021122 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:17.022593 27572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 16:43:17.022790 27572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "c1f9900226704f7a956d88bd72fa262b"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "11e0bac2d0c59c6515304a83fabc8fdc"
server_key_iv: "1894ca53036bbbb787891ec256303b6e"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.023003 27572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:17.041414 27572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:17.045760 27572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:17.046046 27572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:17.063714 27572 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45605
I20260928 16:43:17.063731 27638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45605 every 8 connection(s)
I20260928 16:43:17.065160 27572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:17.069128 27639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:17.072794 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27572
I20260928 16:43:17.073043 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:17.073851 27028 external_mini_cluster.cc:1428] Setting key 3bca90e8faefb64f3f1a60a9d096a5f6
I20260928 16:43:17.082816 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b: Bootstrap starting.
I20260928 16:43:17.086764 27639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:17.088189 27639 log.cc:824] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:43:17 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:17.091599 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b: No bootstrap required, opened a new log
I20260928 16:43:17.097886 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.076090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.26.101.62:45605)
0928 16:43:17.076889 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:17.076910 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:17.076954 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:43:17.078636 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:17.078664 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:17.079193 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:17.079568 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:17.081425 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.082746 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.084497 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.085243 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.085374 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.090215 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:17.090223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:17.090239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:17.090278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:17.093440 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.094334 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.094344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.094351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.094539 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.094925 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.094929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.094931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.095572 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:17.095786 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:17.096343 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 16:43:17.096761 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":359,"threads_started":1}
I20260928 16:43:17.100767 27639 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } }
I20260928 16:43:17.101416 27639 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:17.101514 27639 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f9900226704f7a956d88bd72fa262b, State: Initialized, Role: FOLLOWER
I20260928 16:43:17.102115 27639 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } }
I20260928 16:43:17.102357 27639 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:17.102463 27639 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:17.102592 27639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:17.104833 27639 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } }
I20260928 16:43:17.105434 27639 leader_election.cc:302] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f9900226704f7a956d88bd72fa262b; no voters:
I20260928 16:43:17.106091 27639 leader_election.cc:288] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:17.106221 27644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:17.106865 27644 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [term 1 LEADER]: Becoming Leader. State: Replica: c1f9900226704f7a956d88bd72fa262b, State: Running, Role: LEADER
I20260928 16:43:17.107486 27644 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } }
I20260928 16:43:17.108116 27639 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:17.109905 27646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f9900226704f7a956d88bd72fa262b. Latest consensus state: current_term: 1 leader_uuid: "c1f9900226704f7a956d88bd72fa262b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } } }
I20260928 16:43:17.110348 27646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:17.111974 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f9900226704f7a956d88bd72fa262b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f9900226704f7a956d88bd72fa262b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45605 } } }
I20260928 16:43:17.112411 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:17.114073 27650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:17.119724 27650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:17.130178 27650 catalog_manager.cc:1347] Generated new cluster ID: 23fc06b2a9c64cd0806c08796606d459
I20260928 16:43:17.130297 27650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:17.142549 27650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:17.144155 27650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:17.152294 27650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1f9900226704f7a956d88bd72fa262b: Generated new TSK 0
I20260928 16:43:17.153335 27650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:17.210870 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:17.624284 27665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:17.624622 27665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:17.624779 27665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:17.632980 27665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:17.633106 27665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:17.633266 27665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:17.643780 27665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:17.645066 27665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:17.646505 27665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:17.661578 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:17.661557 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:17.661618 27673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:17.662770 27665 server_base.cc:1034] running on GCE node
I20260928 16:43:17.663522 27665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:17.664659 27665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:17.666028 27665 hybrid_clock.cc:648] HybridClock initialized: now 1790613797665934 us; error 63 us; skew 500 ppm
I20260928 16:43:17.666479 27665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:17 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613797, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:17.670663 27665 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:17.672626 27665 webserver.cc:460] Webserver started at http://127.26.101.1:46147/ using document root <none> and password file <none>
I20260928 16:43:17.673385 27665 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:17.673507 27665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:17.673859 27665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:17.677318 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "d129da24dea04bcb946fa9433a75c880"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "4572e38e4f472cbc15e77715161ff18d"
server_key_iv: "e506c983bbe6818078bbcb0c71faaa00"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.678151 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "d129da24dea04bcb946fa9433a75c880"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "4572e38e4f472cbc15e77715161ff18d"
server_key_iv: "e506c983bbe6818078bbcb0c71faaa00"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.683923 27665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:43:17.688203 27680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:17.689909 27665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:17.690120 27665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "d129da24dea04bcb946fa9433a75c880"
format_stamp: "Formatted at 2026-09-28 16:43:17 on dist-test-slave-t0tw"
server_key: "4572e38e4f472cbc15e77715161ff18d"
server_key_iv: "e506c983bbe6818078bbcb0c71faaa00"
server_key_version: "encryptionkey@0"
I20260928 16:43:17.690351 27665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:17.726112 27665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:17.731359 27665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:17.731662 27665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:17.733098 27665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:17.735001 27665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:17.735116 27665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:43:17.735256 27665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:17.735317 27665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:17.776876 27665 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:45801
I20260928 16:43:17.776883 27792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:45801 every 8 connection(s)
I20260928 16:43:17.778347 27665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:17.782553 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27665
I20260928 16:43:17.782789 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:17.783725 27028 external_mini_cluster.cc:1428] Setting key 6f58c9a4656d06963fcd5d3f3c35dba7
Sep 28 16:43:17 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613797, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:43:17.802507 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.781207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:38687 (local address 127.26.101.62:45605)
0928 16:43:17.781658 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:17.781664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:17.783173 (+ 1509us) server_negotiation.cc:407] Connection header received
0928 16:43:17.785269 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:17.785273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:17.785340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:17.785424 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:17.787571 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.788434 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.790573 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.791290 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.791427 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.795665 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:17.795673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:17.795678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:17.795723 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:17.799037 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.799996 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.800001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.800005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.800159 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:17.800622 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:17.800628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:17.800631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:17.801031 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:17.801188 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:17.801964 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 16:43:17.802141 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260928 16:43:17.803489 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:17.781734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45605 (local address 127.26.101.1:38687)
0928 16:43:17.782585 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:17.782641 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:17.784943 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:17.785692 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:17.785700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:17.786437 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:17.787382 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.787412 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.788602 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.788607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.790431 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:17.790446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:17.791570 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:17.791577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:17.791871 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:17.792650 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:17.792690 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:17.795408 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:17.799214 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.799221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.799237 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.799824 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.800286 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:17.800289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:17.800291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:17.800494 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:17.801173 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:17.801179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:17.801565 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 16:43:17.801835 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 16:43:17.802267 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:43:17.805455 27793 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45605
I20260928 16:43:17.805945 27793 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:17.806780 27793 heartbeater.cc:511] Master 127.26.101.62:45605 requested a full tablet report, sending...
I20260928 16:43:17.809279 27604 ts_manager.cc:194] Registered new tserver with Master: d129da24dea04bcb946fa9433a75c880 (127.26.101.1:45801)
I20260928 16:43:17.811348 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:38687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:17.828445 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:18.231197 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:18.231519 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:18.231688 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:18.239773 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:18.239992 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:18.240137 27799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:18.252084 27799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:18.253556 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:18.255035 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:18.270036 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:18.270412 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:18.270991 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:18.271654 27799 server_base.cc:1034] running on GCE node
I20260928 16:43:18.272516 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:18.273764 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:18.275007 27799 hybrid_clock.cc:648] HybridClock initialized: now 1790613798274930 us; error 66 us; skew 500 ppm
I20260928 16:43:18.275398 27799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:18 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613798, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:18.279273 27799 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:18.281517 27799 webserver.cc:460] Webserver started at http://127.26.101.2:42975/ using document root <none> and password file <none>
I20260928 16:43:18.282369 27799 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:18.282512 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:18.282836 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:18.286600 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "7ba0b3c13e884a2b8251782cabb80b98"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "e3a96b5a28e9a71439c44062d5cb6fd5"
server_key_iv: "79b9ca4d83235105f84817010b4b473f"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.287945 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "7ba0b3c13e884a2b8251782cabb80b98"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "e3a96b5a28e9a71439c44062d5cb6fd5"
server_key_iv: "79b9ca4d83235105f84817010b4b473f"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.294934 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:43:18.299369 27814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.301203 27799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:43:18.301397 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "7ba0b3c13e884a2b8251782cabb80b98"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "e3a96b5a28e9a71439c44062d5cb6fd5"
server_key_iv: "79b9ca4d83235105f84817010b4b473f"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.301618 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:18.326880 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:18.331797 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:18.332098 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:18.333211 27799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:18.334756 27799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:18.334846 27799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.334982 27799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:18.335038 27799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.368296 27799 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:38465
I20260928 16:43:18.368348 27926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:38465 every 8 connection(s)
I20260928 16:43:18.369623 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:18.378080 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27799
I20260928 16:43:18.378296 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:18.378821 27028 external_mini_cluster.cc:1428] Setting key c983417002c38d3e13ee6a48ffe145ff
Sep 28 16:43:18 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613798, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:18.392660 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.372323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:34981 (local address 127.26.101.62:45605)
0928 16:43:18.372758 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.372763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.373854 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 16:43:18.375810 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.375817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.375907 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.376007 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:18.377679 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.378737 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.380734 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.381290 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.381400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.385991 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:18.385997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:18.386000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:18.386041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:18.389335 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.390194 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.390200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.390204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.390375 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.390761 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.390765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.390768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.391168 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:18.391320 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.392052 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.392290 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260928 16:43:18.393729 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.372924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45605 (local address 127.26.101.2:34981)
0928 16:43:18.373643 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.373687 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:18.375493 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:18.376266 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:18.376275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:18.376845 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:18.377479 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.377499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.378975 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.378980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.380538 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.380556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.381558 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.381567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.381911 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.382701 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:18.382731 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:18.385658 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:18.389519 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:18.389527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:18.389542 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:18.390052 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:18.390469 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:18.390473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:18.390475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:18.390665 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:18.391295 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:18.391301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:18.391672 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 16:43:18.391904 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 16:43:18.392372 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":280,"threads_started":1}
I20260928 16:43:18.395620 27927 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45605
I20260928 16:43:18.396055 27927 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:18.397131 27927 heartbeater.cc:511] Master 127.26.101.62:45605 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:43:18.398989 27604 ts_manager.cc:194] Registered new tserver with Master: 7ba0b3c13e884a2b8251782cabb80b98 (127.26.101.2:38465)
I20260928 16:43:18.399889 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:34981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:18.422518 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:18.744504 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:18.744814 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:18.744948 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:18.752089 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:18.752230 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:18.752383 27933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:18.762257 27933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:18.763649 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:18.765116 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:18.778415 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:18.778911 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:18.778908 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:18.780021 27933 server_base.cc:1034] running on GCE node
I20260928 16:43:18.780704 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:18.781734 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:18.782938 27933 hybrid_clock.cc:648] HybridClock initialized: now 1790613798782882 us; error 53 us; skew 500 ppm
I20260928 16:43:18.783293 27933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:18 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613798, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:18.786412 27933 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:18.787973 27933 webserver.cc:460] Webserver started at http://127.26.101.3:42535/ using document root <none> and password file <none>
I20260928 16:43:18.788703 27933 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:18.788823 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:18.789122 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:18.792122 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "66878e6c5fca4469947a4739e3711a19"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "2015c1686249c96588332c708d32208e"
server_key_iv: "169d433342bac9b78ea53616f35195ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.792901 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "66878e6c5fca4469947a4739e3711a19"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "2015c1686249c96588332c708d32208e"
server_key_iv: "169d433342bac9b78ea53616f35195ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.798666 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:18.802470 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.803973 27933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:18.804148 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "66878e6c5fca4469947a4739e3711a19"
format_stamp: "Formatted at 2026-09-28 16:43:18 on dist-test-slave-t0tw"
server_key: "2015c1686249c96588332c708d32208e"
server_key_iv: "169d433342bac9b78ea53616f35195ea"
server_key_version: "encryptionkey@0"
I20260928 16:43:18.804369 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:18.815042 27793 heartbeater.cc:503] Master 127.26.101.62:45605 was elected leader, sending a full tablet report...
I20260928 16:43:18.825697 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:18.830426 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:18.830716 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:18.831794 27933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:18.833225 27933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:18.833304 27933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.833415 27933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:18.833469 27933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:18.864475 27933 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:32961
I20260928 16:43:18.864566 28060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:32961 every 8 connection(s)
I20260928 16:43:18.865773 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:18.869814 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 27933
I20260928 16:43:18.870023 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:18.870489 27028 external_mini_cluster.cc:1428] Setting key 0a3feb424863e34fa219065aa7180aa4
Sep 28 16:43:18 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613798, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:18.885914 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.868475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:52567 (local address 127.26.101.62:45605)
0928 16:43:18.868643 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.868648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.869899 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 16:43:18.872053 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.872059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.872149 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.872234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:18.873890 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.874935 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.876518 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.876928 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.877039 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.880436 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:18.880441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:18.880444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:18.880481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:18.882857 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.883625 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.883629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.883632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.883753 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.884180 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.884183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.884185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.884471 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:18.884635 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.885353 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.885587 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:43:18.886889 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.868924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45605 (local address 127.26.101.3:52567)
0928 16:43:18.869701 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.869742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:18.871742 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:18.872499 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:18.872507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:18.873062 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:18.873672 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.873690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.875104 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.875110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.876377 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.876389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.877154 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.877162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.877411 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.878029 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:18.878053 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:18.880212 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:18.882998 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:18.883007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:18.883026 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:18.883503 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:18.883864 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:18.883869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:18.883873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:18.884065 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:18.884600 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:18.884609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:18.884969 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 16:43:18.885209 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 16:43:18.885771 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260928 16:43:18.888267 28061 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45605
I20260928 16:43:18.888590 28061 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:18.889375 28061 heartbeater.cc:511] Master 127.26.101.62:45605 requested a full tablet report, sending...
I20260928 16:43:18.890821 27604 ts_manager.cc:194] Registered new tserver with Master: 66878e6c5fca4469947a4739e3711a19 (127.26.101.3:32961)
I20260928 16:43:18.891661 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:52567
I20260928 16:43:18.898028 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:18.921465 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.908065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.26.101.62:45605)
0928 16:43:18.908268 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.908273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.908680 (+ 407us) server_negotiation.cc:407] Connection header received
0928 16:43:18.908809 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.908814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.908928 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.909039 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:18.910716 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.911553 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.912894 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.913365 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.913475 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.915869 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:18.915874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:18.915878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:18.915925 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:18.918571 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.919342 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.919347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.919350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.919543 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:18.919962 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:18.919967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:18.919971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:18.920433 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:18.920668 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.920948 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.921127 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:43:18.929849 27604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:18.944186 27604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:18.986629 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.965513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45801 (local address 127.0.0.1:33804)
0928 16:43:18.977940 (+ 12427us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.978010 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:18.978370 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:18.979629 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:18.979638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:18.979677 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:18.980175 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.980190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.982120 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.982125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.984167 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.984180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.985563 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.985570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.985807 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.985865 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 16:43:18.985988 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:43:18.986204 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12113,"thread_start_us":11968,"threads_started":1}
I20260928 16:43:18.987294 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.965711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.26.101.1:45801)
0928 16:43:18.977006 (+ 11295us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.977017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.978167 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 16:43:18.978677 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.978688 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.979018 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.979220 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:18.980401 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.981977 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.984328 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.985324 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.985455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.985704 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.986064 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.986235 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11060,"thread_start_us":11312,"threads_started":1}
I20260928 16:43:18.989342 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.965919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.26.101.3:32961)
0928 16:43:18.980830 (+ 14911us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.980839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.980875 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:43:18.980974 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.980981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.981264 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.982213 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:18.982531 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.984175 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.986716 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.988240 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.988374 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.988612 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.988814 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.988993 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14682,"thread_start_us":14527,"threads_started":1}
I20260928 16:43:18.989351 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.965549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:32961 (local address 127.0.0.1:35652)
0928 16:43:18.980023 (+ 14474us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.980042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:18.980173 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:18.981511 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:18.981519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:18.981545 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:18.982005 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.982019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.984341 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.984347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.986513 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.986528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.988443 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.988449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.988617 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.988650 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:43:18.988750 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:43:18.988894 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14354,"mutex_wait_us":23,"thread_start_us":14183,"threads_started":1}
I20260928 16:43:18.990384 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.981132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:38465 (local address 127.0.0.1:41736)
0928 16:43:18.981696 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:18.981713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:18.981840 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:18.982668 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:18.982673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:18.982697 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:18.983126 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.983140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.985017 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.985022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.987221 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:18.987238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.989372 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.989379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:18.989692 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.989727 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:43:18.989857 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:43:18.989992 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":1,"thread_start_us":282,"threads_started":1}
I20260928 16:43:18.990389 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:18.980432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.26.101.2:38465)
0928 16:43:18.981088 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:18.981097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:18.981831 (+ 734us) server_negotiation.cc:407] Connection header received
0928 16:43:18.982008 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:18.982015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:18.982241 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:18.982449 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:18.983337 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.984823 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.987420 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:18.988871 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:18.989000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:18.989184 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:18.989945 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 16:43:18.990113 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":378,"threads_started":1}
I20260928 16:43:18.991827 27728 tablet_service.cc:1467] Processing CreateTablet for tablet b899e62832e746e8b7d95e557621d9af (DEFAULT_TABLE table=test-table [id=f443dc602ccf482cab5b69c74a7d49f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:18.993466 27728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899e62832e746e8b7d95e557621d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:18.994169 27862 tablet_service.cc:1467] Processing CreateTablet for tablet b899e62832e746e8b7d95e557621d9af (DEFAULT_TABLE table=test-table [id=f443dc602ccf482cab5b69c74a7d49f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:18.994184 27996 tablet_service.cc:1467] Processing CreateTablet for tablet b899e62832e746e8b7d95e557621d9af (DEFAULT_TABLE table=test-table [id=f443dc602ccf482cab5b69c74a7d49f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:18.995801 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899e62832e746e8b7d95e557621d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:18.995801 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b899e62832e746e8b7d95e557621d9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:19.006428 28079 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Bootstrap starting.
I20260928 16:43:19.010071 28079 tablet_bootstrap.cc:654] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:19.011173 28079 log.cc:824] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:19.011857 28080 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Bootstrap starting.
I20260928 16:43:19.013576 28081 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Bootstrap starting.
I20260928 16:43:19.016242 28079 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: No bootstrap required, opened a new log
I20260928 16:43:19.016610 28079 ts_tablet_manager.cc:1397] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 16:43:19.016875 28080 tablet_bootstrap.cc:654] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:19.017874 28081 tablet_bootstrap.cc:654] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:19.017916 28080 log.cc:824] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:19.019117 28081 log.cc:824] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:19.020864 28080 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: No bootstrap required, opened a new log
I20260928 16:43:19.021116 28080 ts_tablet_manager.cc:1397] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 16:43:19.022186 28081 tablet_bootstrap.cc:492] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: No bootstrap required, opened a new log
I20260928 16:43:19.022465 28081 ts_tablet_manager.cc:1397] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 16:43:19.027001 28079 raft_consensus.cc:348] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.027748 28079 raft_consensus.cc:374] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:19.027915 28079 raft_consensus.cc:729] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d129da24dea04bcb946fa9433a75c880, State: Initialized, Role: FOLLOWER
I20260928 16:43:19.028676 28079 consensus_queue.cc:260] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.028715 28080 raft_consensus.cc:348] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.029114 28080 raft_consensus.cc:374] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:19.029212 28080 raft_consensus.cc:729] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba0b3c13e884a2b8251782cabb80b98, State: Initialized, Role: FOLLOWER
I20260928 16:43:19.029973 28080 consensus_queue.cc:260] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.030622 28079 ts_tablet_manager.cc:1428] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260928 16:43:19.030896 27794 tablet.cc:2369] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:19.032349 28081 raft_consensus.cc:348] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.032699 28081 raft_consensus.cc:374] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:19.032840 28081 raft_consensus.cc:729] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66878e6c5fca4469947a4739e3711a19, State: Initialized, Role: FOLLOWER
I20260928 16:43:19.032863 28080 ts_tablet_manager.cc:1428] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260928 16:43:19.033098 27927 heartbeater.cc:503] Master 127.26.101.62:45605 was elected leader, sending a full tablet report...
I20260928 16:43:19.033576 28081 consensus_queue.cc:260] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.034775 28061 heartbeater.cc:503] Master 127.26.101.62:45605 was elected leader, sending a full tablet report...
I20260928 16:43:19.035645 28081 ts_tablet_manager.cc:1428] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260928 16:43:19.117687 28062 tablet.cc:2369] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:19.121642 27928 tablet.cc:2369] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:19.241611 28085 raft_consensus.cc:482] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:19.241911 28085 raft_consensus.cc:504] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.243695 28085 leader_election.cc:288] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66878e6c5fca4469947a4739e3711a19 (127.26.101.3:32961), 7ba0b3c13e884a2b8251782cabb80b98 (127.26.101.2:38465)
I20260928 16:43:19.250145 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.244137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53901 (local address 127.26.101.3:32961)
0928 16:43:19.244382 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.244388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.244744 (+ 356us) server_negotiation.cc:407] Connection header received
0928 16:43:19.244963 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.244970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.245059 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.245153 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:19.245802 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.246523 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.248443 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.249253 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.249415 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.249492 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.249777 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.249907 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:43:19.250316 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.244021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:32961 (local address 127.26.101.1:53901)
0928 16:43:19.244597 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:19.244621 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:19.244800 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:19.245251 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:19.245255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:19.245283 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:19.245643 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:19.245660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.246659 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.246663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:19.248281 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:19.248298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.249470 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.249476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:19.249619 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.249648 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:43:19.249734 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:43:19.249884 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":51,"spinlock_wait_cycles":63360,"thread_start_us":289,"threads_started":1}
I20260928 16:43:19.250376 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.245033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45985 (local address 127.26.101.2:38465)
0928 16:43:19.245236 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.245242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.245416 (+ 174us) server_negotiation.cc:407] Connection header received
0928 16:43:19.245628 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.245631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.245689 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.245759 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:19.246267 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.246941 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.248717 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.249493 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.249633 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.249708 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.250009 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.250135 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:43:19.250497 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.244722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:38465 (local address 127.26.101.1:45985)
0928 16:43:19.245316 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:19.245331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:19.245488 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:19.245798 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:19.245802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:19.245825 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:19.246120 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:19.246129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.247098 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.247102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:19.248555 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:19.248567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.249703 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.249709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:19.249853 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.249876 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:43:19.249954 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:43:19.250080 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":34,"thread_start_us":298,"threads_started":1}
I20260928 16:43:19.251351 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899e62832e746e8b7d95e557621d9af" candidate_uuid: "d129da24dea04bcb946fa9433a75c880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66878e6c5fca4469947a4739e3711a19" is_pre_election: true
I20260928 16:43:19.251587 27882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899e62832e746e8b7d95e557621d9af" candidate_uuid: "d129da24dea04bcb946fa9433a75c880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba0b3c13e884a2b8251782cabb80b98" is_pre_election: true
I20260928 16:43:19.251976 28016 raft_consensus.cc:2454] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d129da24dea04bcb946fa9433a75c880 in term 0.
I20260928 16:43:19.252121 27882 raft_consensus.cc:2454] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d129da24dea04bcb946fa9433a75c880 in term 0.
I20260928 16:43:19.252700 27682 leader_election.cc:302] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66878e6c5fca4469947a4739e3711a19, d129da24dea04bcb946fa9433a75c880; no voters:
I20260928 16:43:19.253082 28085 raft_consensus.cc:2781] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:19.253211 28085 raft_consensus.cc:482] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:19.253309 28085 raft_consensus.cc:3037] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:19.255615 28085 raft_consensus.cc:504] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.256243 28085 leader_election.cc:288] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [CANDIDATE]: Term 1 election: Requested vote from peers 66878e6c5fca4469947a4739e3711a19 (127.26.101.3:32961), 7ba0b3c13e884a2b8251782cabb80b98 (127.26.101.2:38465)
I20260928 16:43:19.256852 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899e62832e746e8b7d95e557621d9af" candidate_uuid: "d129da24dea04bcb946fa9433a75c880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66878e6c5fca4469947a4739e3711a19"
I20260928 16:43:19.256898 27882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b899e62832e746e8b7d95e557621d9af" candidate_uuid: "d129da24dea04bcb946fa9433a75c880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba0b3c13e884a2b8251782cabb80b98"
I20260928 16:43:19.257112 27882 raft_consensus.cc:3037] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:19.257121 28016 raft_consensus.cc:3037] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:19.260455 28016 raft_consensus.cc:2454] T b899e62832e746e8b7d95e557621d9af P 66878e6c5fca4469947a4739e3711a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d129da24dea04bcb946fa9433a75c880 in term 1.
I20260928 16:43:19.260455 27882 raft_consensus.cc:2454] T b899e62832e746e8b7d95e557621d9af P 7ba0b3c13e884a2b8251782cabb80b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d129da24dea04bcb946fa9433a75c880 in term 1.
I20260928 16:43:19.260972 27682 leader_election.cc:302] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ba0b3c13e884a2b8251782cabb80b98, d129da24dea04bcb946fa9433a75c880; no voters:
I20260928 16:43:19.261282 28085 raft_consensus.cc:2781] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:19.261740 28085 raft_consensus.cc:686] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [term 1 LEADER]: Becoming Leader. State: Replica: d129da24dea04bcb946fa9433a75c880, State: Running, Role: LEADER
I20260928 16:43:19.262270 28085 consensus_queue.cc:237] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } }
I20260928 16:43:19.270736 27603 catalog_manager.cc:5495] T b899e62832e746e8b7d95e557621d9af P d129da24dea04bcb946fa9433a75c880 reported cstate change: term changed from 0 to 1, leader changed from <none> to d129da24dea04bcb946fa9433a75c880 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "d129da24dea04bcb946fa9433a75c880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d129da24dea04bcb946fa9433a75c880" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66878e6c5fca4469947a4739e3711a19" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ba0b3c13e884a2b8251782cabb80b98" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38465 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:19.360905 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.346022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.26.101.1:45801)
0928 16:43:19.346241 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.346249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.346620 (+ 371us) server_negotiation.cc:407] Connection header received
0928 16:43:19.346794 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.346797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.346867 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.346962 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.348491 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.349224 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.350515 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.350928 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.351052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.355204 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.355223 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.355239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.355279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.357975 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.358708 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.358716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.358718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.358873 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.359309 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.359313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.359315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.359707 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.360035 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.360314 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.360583 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:19.409724 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.395165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.26.101.1:45801)
0928 16:43:19.395389 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.395395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.395686 (+ 291us) server_negotiation.cc:407] Connection header received
0928 16:43:19.395874 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.395880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.395962 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.396069 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.397466 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.398198 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.399522 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.399946 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.400058 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.404206 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.404212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.404215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.404264 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.406928 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.407629 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.407634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.407636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.407783 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.408253 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.408256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.408258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.408587 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.408868 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.409145 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.409375 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:19.455331 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.440839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.26.101.1:45801)
0928 16:43:19.441062 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.441069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.441326 (+ 257us) server_negotiation.cc:407] Connection header received
0928 16:43:19.441507 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.441511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.441594 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.441688 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.442979 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.443793 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.445034 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.445474 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.445588 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.449796 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.449802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.449805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.449847 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.452471 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.453210 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.453215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.453217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.453363 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.453744 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.453747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.453749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.454052 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.454420 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.454702 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.454933 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 16:43:19.457204 27708 tablet_service.cc:2908] Rejecting scan request for tablet b899e62832e746e8b7d95e557621d9af: Uninitialized: clean time has not yet been initialized
I20260928 16:43:19.462358 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27665
I20260928 16:43:19.478581 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.463372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.26.101.1:45801)
0928 16:43:19.463566 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.463573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.463884 (+ 311us) server_negotiation.cc:407] Connection header received
0928 16:43:19.464071 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.464077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.464147 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.464248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.465473 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.466380 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.467722 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.468218 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.468363 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.471349 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.471357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.471362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.471414 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.475111 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.476127 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.476134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.476137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.476330 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.476856 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.476862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.476865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.477197 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.477468 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.477919 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.478191 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:43:19.479566 27788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:19.578255 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27665
I20260928 16:43:19.594628 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27799
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:19.609902 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.595397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.26.101.2:38465)
0928 16:43:19.595573 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.595580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.595625 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:43:19.595831 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.595835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.595899 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.596012 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.597216 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.597926 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.599247 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.599642 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.599756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.603828 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.603844 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.603861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.603900 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.606863 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.607595 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.607603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.607606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.607815 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.608204 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.608208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.608210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.608710 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.609012 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.609313 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.609559 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:43:19.610514 27922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:19.698772 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27799
I20260928 16:43:19.714919 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27933
Sep 28 16:43:19 dist-test-slave-t0tw krb5kdc[27559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:19.731447 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:19.715796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.26.101.3:32961)
0928 16:43:19.716011 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:19.716017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:19.716056 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:43:19.716181 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:19.716192 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:19.716280 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:19.716386 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:19.717962 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.718729 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.720171 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:19.720563 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:19.720661 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:19.724986 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:19.725004 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:19.725022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:19.725063 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:19.727984 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.728828 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.728836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.728839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.729010 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:19.729551 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:19.729556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:19.729558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:19.730007 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:19.730261 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:19.730630 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:43:19.730825 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:43:19.732282 28056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:19.823123 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27933
I20260928 16:43:19.838368 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 27572
I20260928 16:43:19.839574 27634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:19.949469 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 27572
2026-09-28T16:43:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4480 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:20 dist-test-slave-t0tw krb5kdc[28109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:20 dist-test-slave-t0tw krb5kdc[28109](info): set up 2 sockets
Sep 28 16:43:20 dist-test-slave-t0tw krb5kdc[28109](info): commencing operation
krb5kdc: starting...
W20260928 16:43:20.954684 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:21 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:21.139300 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:43261
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:21.454286 28122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:21.454610 28122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:21.454747 28122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:21.461944 28122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:21.462066 28122 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:43:21.462137 28122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:21.462224 28122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:21.462291 28122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:21.462374 28122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:21.472420 28122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:43261
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:21.473695 28122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:21.475018 28122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:21.486390 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:21.486434 28128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:21.486455 28127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:21.487221 28122 server_base.cc:1034] running on GCE node
I20260928 16:43:21.487967 28122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:21.489331 28122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:21.490564 28122 hybrid_clock.cc:648] HybridClock initialized: now 1790613801490507 us; error 27 us; skew 500 ppm
I20260928 16:43:21.490903 28122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:21 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613801, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:21.493974 28122 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:21.495466 28122 webserver.cc:460] Webserver started at http://127.26.101.62:44301/ using document root <none> and password file <none>
I20260928 16:43:21.496165 28122 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:21.496268 28122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:21.496570 28122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:21.499547 28122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "58f60977939e45ff8139dbda7a0b453b"
format_stamp: "Formatted at 2026-09-28 16:43:21 on dist-test-slave-t0tw"
server_key: "daf45795620e70a107f4b85adec6c75c"
server_key_iv: "1bd792666c469e322952adedf4f37910"
server_key_version: "encryptionkey@0"
I20260928 16:43:21.500308 28122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "58f60977939e45ff8139dbda7a0b453b"
format_stamp: "Formatted at 2026-09-28 16:43:21 on dist-test-slave-t0tw"
server_key: "daf45795620e70a107f4b85adec6c75c"
server_key_iv: "1bd792666c469e322952adedf4f37910"
server_key_version: "encryptionkey@0"
I20260928 16:43:21.505623 28122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:21.509172 28137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:21.510531 28122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:21.510697 28122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "58f60977939e45ff8139dbda7a0b453b"
format_stamp: "Formatted at 2026-09-28 16:43:21 on dist-test-slave-t0tw"
server_key: "daf45795620e70a107f4b85adec6c75c"
server_key_iv: "1bd792666c469e322952adedf4f37910"
server_key_version: "encryptionkey@0"
I20260928 16:43:21.510906 28122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:21.525676 28122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:21.529980 28122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:21.530239 28122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:21.550459 28122 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:43261
I20260928 16:43:21.550479 28199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:43261 every 8 connection(s)
I20260928 16:43:21.551882 28122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:21.555764 28200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:21.555974 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28122
I20260928 16:43:21.556205 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:21.557039 27028 external_mini_cluster.cc:1428] Setting key f0de7dbf48245a8b2dde9270f4eced76
I20260928 16:43:21.569690 28200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Bootstrap starting.
Sep 28 16:43:21 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:21.573551 28200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:21.574684 28200 log.cc:824] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:21.578030 28200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: No bootstrap required, opened a new log
I20260928 16:43:21.582226 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:21.559178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.26.101.62:43261)
0928 16:43:21.559961 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:21.559973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:21.560014 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:43:21.562032 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:21.562055 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:21.562480 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:21.562831 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:21.564574 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:21.565987 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:21.567812 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:21.568519 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:21.568653 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:21.573419 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:21.573431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:21.573451 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:21.573492 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:21.577092 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:21.577929 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:21.577938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:21.577946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:21.578180 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:21.578602 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:21.578607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:21.578610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:21.579429 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:21.579727 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:21.580117 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 16:43:21.580607 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":340,"threads_started":1}
I20260928 16:43:21.587767 28200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } }
I20260928 16:43:21.588142 28200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:21.588227 28200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f60977939e45ff8139dbda7a0b453b, State: Initialized, Role: FOLLOWER
I20260928 16:43:21.588721 28200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } }
I20260928 16:43:21.588953 28200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:21.589092 28200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:21.589267 28200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:21.591753 28200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } }
I20260928 16:43:21.592263 28200 leader_election.cc:302] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58f60977939e45ff8139dbda7a0b453b; no voters:
I20260928 16:43:21.592806 28200 leader_election.cc:288] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:21.592929 28205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:21.593955 28205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [term 1 LEADER]: Becoming Leader. State: Replica: 58f60977939e45ff8139dbda7a0b453b, State: Running, Role: LEADER
I20260928 16:43:21.594540 28205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } }
I20260928 16:43:21.594902 28200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:21.597110 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f60977939e45ff8139dbda7a0b453b. Latest consensus state: current_term: 1 leader_uuid: "58f60977939e45ff8139dbda7a0b453b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } } }
I20260928 16:43:21.597486 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:21.597988 28206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58f60977939e45ff8139dbda7a0b453b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f60977939e45ff8139dbda7a0b453b" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 43261 } } }
I20260928 16:43:21.598260 28206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:21.598551 28213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:21.604737 28213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:21.614292 28213 catalog_manager.cc:1347] Generated new cluster ID: da3f36c6947249668c45788698d3606f
I20260928 16:43:21.614415 28213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:21.627920 28213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:21.629417 28213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:21.637298 28213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Generated new TSK 0
I20260928 16:43:21.638438 28213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:21.687525 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:43261
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:22.011945 28226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:22.012248 28226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:22.012389 28226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:22.019729 28226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:22.019851 28226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:22.019987 28226 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:22.020095 28226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:22.030038 28226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:22.031458 28226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:22.032888 28226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:22.046640 28234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:22.046674 28232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:22.046698 28231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:22.047485 28226 server_base.cc:1034] running on GCE node
I20260928 16:43:22.048166 28226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:22.049172 28226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:22.050402 28226 hybrid_clock.cc:648] HybridClock initialized: now 1790613802050336 us; error 46 us; skew 500 ppm
I20260928 16:43:22.050704 28226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:22 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613802, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:22.054039 28226 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:22.055622 28226 webserver.cc:460] Webserver started at http://127.26.101.1:39159/ using document root <none> and password file <none>
I20260928 16:43:22.056367 28226 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:22.056466 28226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:22.056793 28226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:22.059923 28226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "5dc2fb5c62014dc5970163c212585a7a"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "09847679ae3fcf6a6f28338167c816d1"
server_key_iv: "7a980608a842b7714e89cb6753fe0ec0"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.060782 28226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc2fb5c62014dc5970163c212585a7a"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "09847679ae3fcf6a6f28338167c816d1"
server_key_iv: "7a980608a842b7714e89cb6753fe0ec0"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.066464 28226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:22.070226 28241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:22.071739 28226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:43:22.071913 28226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "5dc2fb5c62014dc5970163c212585a7a"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "09847679ae3fcf6a6f28338167c816d1"
server_key_iv: "7a980608a842b7714e89cb6753fe0ec0"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.072140 28226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:22.093569 28226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:22.098127 28226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:22.098387 28226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:22.101339 28226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:22.101454 28226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:22.101603 28226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:22.101672 28226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:22 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613802, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:22.127976 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.102114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:48745 (local address 127.26.101.62:43261)
0928 16:43:22.108072 (+ 5958us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:22.108077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:22.108101 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:43:22.108155 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:22.108158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:22.108219 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:22.108315 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:22.110951 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.111911 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.115048 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.115749 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.115895 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.120255 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:22.120263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:22.120268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:22.120318 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:22.123725 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.124832 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.124838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.124841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.125001 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.125726 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.125731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.125734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.126145 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:22.126326 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:22.127353 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0928 16:43:22.127591 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5842,"thread_start_us":5730,"threads_started":1}
I20260928 16:43:22.129110 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.102834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.1:48745)
0928 16:43:22.104042 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.104115 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:22.107309 (+ 3194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:22.108665 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:22.108678 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:22.109635 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:22.110686 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.110722 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.112124 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.112130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.114848 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.114868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.116020 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.116028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.116280 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.117033 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:22.117060 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:22.119944 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:22.123887 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.123898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.123927 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.124611 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.125151 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.125160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.125164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.125546 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.126318 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:22.126327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:22.126859 (+ 532us) client_negotiation.cc:769] Sending connection context
0928 16:43:22.127273 (+ 414us) client_negotiation.cc:240] Negotiation successful
0928 16:43:22.127873 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":422,"threads_started":1}
I20260928 16:43:22.150200 28226 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:44193
I20260928 16:43:22.150218 28363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:44193 every 8 connection(s)
I20260928 16:43:22.150731 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.135481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45265 (local address 127.26.101.62:43261)
0928 16:43:22.135648 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:22.135653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:22.136603 (+ 950us) server_negotiation.cc:407] Connection header received
0928 16:43:22.136925 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:22.136931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:22.137021 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:22.137098 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:22.138460 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.139466 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.141431 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.141831 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.141944 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.144215 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:22.144223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:22.144228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:22.144274 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:22.147633 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.148556 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.148560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.148563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.148701 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.149146 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.149151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.149154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.149516 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:22.149660 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:22.150126 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 16:43:22.150348 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:43:22.150852 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.135718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.1:45265)
0928 16:43:22.136481 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.136505 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:22.136737 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:22.137317 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:22.137319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:22.137832 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:22.138308 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.138323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.139659 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.139664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.141274 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.141287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.142047 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.142055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.142199 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.142884 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:22.142901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:22.143921 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:22.147786 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.147801 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.147806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.148434 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.148819 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.148824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.148828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.149015 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.149639 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:22.149644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:22.149842 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 16:43:22.150004 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:43:22.150372 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"spinlock_wait_cycles":31616,"thread_start_us":557,"threads_started":1}
I20260928 16:43:22.151827 28226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:22.154196 28364 heartbeater.cc:347] Connected to a master server at 127.26.101.62:43261
I20260928 16:43:22.154448 28364 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:22.155202 28364 heartbeater.cc:511] Master 127.26.101.62:43261 requested a full tablet report, sending...
I20260928 16:43:22.156085 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28226
I20260928 16:43:22.156306 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:22.157034 27028 external_mini_cluster.cc:1428] Setting key 23ae5c538415e540450219ab4de23cfb
I20260928 16:43:22.168668 28154 ts_manager.cc:194] Registered new tserver with Master: 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193)
I20260928 16:43:22.170185 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:48745
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:22.203547 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:43261
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:22.522431 28368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:22.522770 28368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:22.522925 28368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:22.530228 28368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:22.530362 28368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:22.530490 28368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:22.530607 28368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:22.540625 28368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:22.541981 28368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:22.543358 28368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:22.556679 28373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:22.556658 28374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:22.557291 28376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:22.557967 28368 server_base.cc:1034] running on GCE node
I20260928 16:43:22.558614 28368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:22.559640 28368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:22.560868 28368 hybrid_clock.cc:648] HybridClock initialized: now 1790613802560789 us; error 54 us; skew 500 ppm
I20260928 16:43:22.561170 28368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:22 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613802, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:22.564459 28368 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:22.565984 28368 webserver.cc:460] Webserver started at http://127.26.101.2:45307/ using document root <none> and password file <none>
I20260928 16:43:22.566689 28368 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:22.566792 28368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:22.567128 28368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:22.570132 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "c8e92562b7034c449c0ef0a04617674b"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "862ce3dc08a5dcd159f39b80f2e9a819"
server_key_iv: "6d3024f3defc7eff65c60bf7a6f74919"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.570917 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "c8e92562b7034c449c0ef0a04617674b"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "862ce3dc08a5dcd159f39b80f2e9a819"
server_key_iv: "6d3024f3defc7eff65c60bf7a6f74919"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.576267 28368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:43:22.579777 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:22.581140 28368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:22.581295 28368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "c8e92562b7034c449c0ef0a04617674b"
format_stamp: "Formatted at 2026-09-28 16:43:22 on dist-test-slave-t0tw"
server_key: "862ce3dc08a5dcd159f39b80f2e9a819"
server_key_iv: "6d3024f3defc7eff65c60bf7a6f74919"
server_key_version: "encryptionkey@0"
I20260928 16:43:22.581501 28368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:22.605659 28368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:22.610221 28368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:22.610498 28368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:22.613176 28368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:22.613268 28368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:22.613411 28368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:22.613471 28368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:22 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613802, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:22.638536 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.613868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:47615 (local address 127.26.101.62:43261)
0928 16:43:22.614066 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:22.614072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:22.615600 (+ 1528us) server_negotiation.cc:407] Connection header received
0928 16:43:22.617735 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:22.617742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:22.617841 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:22.617959 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:22.620626 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.621581 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.624549 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.624991 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.625092 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.630972 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:22.630981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:22.630986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:22.631039 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:22.634720 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.635807 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.635814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.635819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.636001 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.636497 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.636502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.636504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.636867 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:22.637029 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:22.638098 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0928 16:43:22.638277 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:43:22.640503 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.614445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.2:47615)
0928 16:43:22.615330 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.615456 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:22.617393 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:22.618282 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:22.618298 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:22.619318 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:22.620380 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.620427 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.621790 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.621795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.624362 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.624395 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.625231 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.625239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.625569 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.626587 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:22.626681 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:22.630562 (+ 3881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:22.634891 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.634914 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.634938 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.635618 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.636119 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.636122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.636124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.636341 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.637032 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:22.637041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:22.637644 (+ 603us) client_negotiation.cc:769] Sending connection context
0928 16:43:22.637979 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 16:43:22.638652 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":369,"threads_started":1}
I20260928 16:43:22.655055 28368 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:33415
I20260928 16:43:22.655069 28503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:33415 every 8 connection(s)
I20260928 16:43:22.656582 28368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:22.658810 28504 heartbeater.cc:347] Connected to a master server at 127.26.101.62:43261
I20260928 16:43:22.659058 28504 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:22.659860 28504 heartbeater.cc:511] Master 127.26.101.62:43261 requested a full tablet report, sending...
I20260928 16:43:22.661379 28154 ts_manager.cc:194] Registered new tserver with Master: c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415)
I20260928 16:43:22.661439 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28368
I20260928 16:43:22.661628 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:22.662150 27028 external_mini_cluster.cc:1428] Setting key ac06c9f6228ff6fb73d9b1aad8c38233
I20260928 16:43:22.662161 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.647727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.2:55853)
0928 16:43:22.648456 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:22.648479 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:22.648659 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:22.648992 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:22.648997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:22.649646 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:22.650058 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.650074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.651044 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.651049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.652974 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:22.652995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.653848 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.653859 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:22.654069 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.654828 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:22.654858 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:22.655990 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:22.659502 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.659506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.659509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.659919 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.660314 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:22.660318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:22.660322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:22.660484 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:22.661104 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:22.661108 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:22.661281 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 16:43:22.661399 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:43:22.661730 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":354,"threads_started":1}
I20260928 16:43:22.662516 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:47615
I20260928 16:43:22.662106 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:22.647710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:55853 (local address 127.26.101.62:43261)
0928 16:43:22.647937 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:22.647944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:22.648584 (+ 640us) server_negotiation.cc:407] Connection header received
0928 16:43:22.648789 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:22.648793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:22.648858 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:22.648935 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:22.650193 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.650898 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.653165 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:22.653626 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:22.653740 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:22.656187 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:22.656193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:22.656196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:22.656229 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:22.659379 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.660058 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.660064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.660068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.660202 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:22.660581 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:22.660585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:22.660587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:22.660982 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:22.661129 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:22.661532 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 16:43:22.661750 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:22.700657 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:43261
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:23.030736 28508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:23.031103 28508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:23.031250 28508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:23.038592 28508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:23.038739 28508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:23.038863 28508 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:23.038975 28508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:23.049026 28508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:43261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:23.050465 28508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:23.051915 28508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:23.065500 28514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:23.065757 28516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:23.065977 28513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:23.066251 28508 server_base.cc:1034] running on GCE node
I20260928 16:43:23.066969 28508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:23.068028 28508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:23.069306 28508 hybrid_clock.cc:648] HybridClock initialized: now 1790613803069230 us; error 38 us; skew 500 ppm
I20260928 16:43:23.069651 28508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:23 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:23.073174 28508 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:23.074769 28508 webserver.cc:460] Webserver started at http://127.26.101.3:40589/ using document root <none> and password file <none>
I20260928 16:43:23.075634 28508 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:23.075747 28508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:23.076107 28508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:23.079316 28508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "1254e1b365d5431ba5cf9c80ef544d02"
format_stamp: "Formatted at 2026-09-28 16:43:23 on dist-test-slave-t0tw"
server_key: "57eae198f8a4af83367b3c3022526bd9"
server_key_iv: "734531164dd390ab6f2c6d5e58cfb47e"
server_key_version: "encryptionkey@0"
I20260928 16:43:23.080286 28508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "1254e1b365d5431ba5cf9c80ef544d02"
format_stamp: "Formatted at 2026-09-28 16:43:23 on dist-test-slave-t0tw"
server_key: "57eae198f8a4af83367b3c3022526bd9"
server_key_iv: "734531164dd390ab6f2c6d5e58cfb47e"
server_key_version: "encryptionkey@0"
I20260928 16:43:23.086045 28508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:23.089854 28523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:23.091298 28508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:23.091462 28508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "1254e1b365d5431ba5cf9c80ef544d02"
format_stamp: "Formatted at 2026-09-28 16:43:23 on dist-test-slave-t0tw"
server_key: "57eae198f8a4af83367b3c3022526bd9"
server_key_iv: "734531164dd390ab6f2c6d5e58cfb47e"
server_key_version: "encryptionkey@0"
I20260928 16:43:23.091681 28508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:23.109144 28508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:23.114034 28508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:23.114363 28508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:23.117324 28508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:23.117442 28508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:23.117597 28508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:23.117671 28508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:23 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:23.144593 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.118015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36091 (local address 127.26.101.62:43261)
0928 16:43:23.118229 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.118236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.119841 (+ 1605us) server_negotiation.cc:407] Connection header received
0928 16:43:23.122158 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.122165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.122261 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.122367 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:23.125470 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.126265 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.129564 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.130275 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.130435 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.136150 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:23.136159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:23.136164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:23.136217 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:23.140038 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.141293 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.141299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.141303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.141458 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.142126 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.142142 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.142146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.142629 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:23.142869 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.143989 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.144236 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:43:23.145686 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.118790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.3:36091)
0928 16:43:23.119670 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.119709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.121841 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.122884 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.122899 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.124031 (+ 1132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:23.125199 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.125237 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.126456 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.126462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.129322 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.129349 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.130595 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.130604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.130958 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.131901 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:23.131941 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:23.135751 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:23.140271 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.140282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.140308 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.141095 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.141578 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.141583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.141587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.141947 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.142797 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:23.142809 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:23.143441 (+ 632us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.143825 (+ 384us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.144432 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":366,"threads_started":1}
I20260928 16:43:23.166129 28508 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:38163
I20260928 16:43:23.166142 28643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:38163 every 8 connection(s)
I20260928 16:43:23.167706 28508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:23.168283 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28508
I20260928 16:43:23.168478 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:23.168999 27028 external_mini_cluster.cc:1428] Setting key 7dc0cbb2d28e85a91c51161a087841f3
I20260928 16:43:23.169878 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.151585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.26.101.3:54099)
0928 16:43:23.154440 (+ 2855us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.154460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.154686 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.155194 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.155199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.156029 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:23.156513 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.156532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.158003 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.158012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.159574 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.159595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.160625 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.160636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.160836 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.161600 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:23.161617 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:23.162830 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:23.166853 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.166859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.166863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.167323 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.167732 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.167736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.167738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.167883 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.168560 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:23.168567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:23.168797 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.168951 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.169369 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2689,"thread_start_us":2542,"threads_started":1}
I20260928 16:43:23.171099 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.151485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:54099 (local address 127.26.101.62:43261)
0928 16:43:23.151660 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.151666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.154562 (+ 2896us) server_negotiation.cc:407] Connection header received
0928 16:43:23.154879 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.154886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.154991 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.155132 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:23.156739 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.157860 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.159750 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.160410 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.160553 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.163068 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:23.163092 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:23.163098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:23.163141 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:23.166715 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.167444 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.167450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.167453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.167625 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.168022 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.168027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.168030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.168406 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:23.170302 (+ 1896us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.170464 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.170712 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:23.171852 28644 heartbeater.cc:347] Connected to a master server at 127.26.101.62:43261
I20260928 16:43:23.172147 28644 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:23.173079 28644 heartbeater.cc:511] Master 127.26.101.62:43261 requested a full tablet report, sending...
I20260928 16:43:23.173429 28364 heartbeater.cc:503] Master 127.26.101.62:43261 was elected leader, sending a full tablet report...
I20260928 16:43:23.174511 28154 ts_manager.cc:194] Registered new tserver with Master: 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163)
I20260928 16:43:23.175432 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:36091
I20260928 16:43:23.184916 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:43:23 dist-test-slave-t0tw krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:43:23 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:23.230163 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.215835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.26.101.62:43261)
0928 16:43:23.216077 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.216083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.216328 (+ 245us) server_negotiation.cc:407] Connection header received
0928 16:43:23.216477 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.216481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.216551 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.216636 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:23.217969 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.218737 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.220051 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.220482 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.220583 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.224744 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:23.224750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:23.224754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:23.224811 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:23.227532 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.228185 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.228189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.228191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.228320 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.228675 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.228680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.228683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.229021 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:23.229239 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.229558 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.229738 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:43:23.234484 28154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:23.241357 28154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:23.279865 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.254810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.26.101.2:33415)
0928 16:43:23.268790 (+ 13980us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.268798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.269036 (+ 238us) server_negotiation.cc:407] Connection header received
0928 16:43:23.269255 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.269261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.269507 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.269739 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.271676 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.273369 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.276513 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.278139 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.278269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.278514 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.279305 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.279515 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13811,"thread_start_us":13667,"threads_started":1}
I20260928 16:43:23.280862 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.253964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:38163 (local address 127.0.0.1:51946)
0928 16:43:23.266021 (+ 12057us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.266085 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.266358 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.271453 (+ 5095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.271459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.271497 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.271957 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.271971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.274611 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.274618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.277466 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.277488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.279840 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.279847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.280051 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.280089 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.280157 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.280407 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11656,"thread_start_us":15239,"threads_started":1}
I20260928 16:43:23.280982 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.254345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33415 (local address 127.0.0.1:35834)
0928 16:43:23.268883 (+ 14538us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.268908 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.269103 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.270812 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.270852 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.270886 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.271407 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.271427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.273596 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.273603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.276266 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.276289 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.278712 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.278719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.278954 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.279014 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.280362 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.280541 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14314,"thread_start_us":12256,"threads_started":1}
I20260928 16:43:23.282773 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.254147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.26.101.3:38163)
0928 16:43:23.270450 (+ 16303us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.270461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.270501 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:43:23.270612 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.270620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.270908 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.272186 (+ 1278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.272551 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.274419 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.277830 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.279567 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.281806 (+ 2239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.282118 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.282198 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.282403 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16091,"thread_start_us":16952,"threads_started":1}
I20260928 16:43:23.285784 28433 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (DEFAULT_TABLE table=test-table [id=3161988dd631432b8a7689fd29985b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.287204 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.287245 28573 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (DEFAULT_TABLE table=test-table [id=3161988dd631432b8a7689fd29985b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.288789 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.291878 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.269717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44193 (local address 127.0.0.1:44848)
0928 16:43:23.281145 (+ 11428us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.281168 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.281296 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.284954 (+ 3658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.284960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.284989 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.285173 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.285186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.287385 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.287391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.289178 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.289195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.291058 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.291064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.291287 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.291338 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.291410 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.291567 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11299,"thread_start_us":13438,"threads_started":1}
I20260928 16:43:23.292488 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.270008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.26.101.1:44193)
0928 16:43:23.283909 (+ 13901us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.283917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.283946 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:43:23.284044 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.284052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.284717 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.285391 (+ 674us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.285684 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.287214 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.289363 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.290840 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.291728 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.291962 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.292035 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.292208 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13708,"thread_start_us":14883,"threads_started":1}
I20260928 16:43:23.296682 28293 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (DEFAULT_TABLE table=test-table [id=3161988dd631432b8a7689fd29985b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.298370 28293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.301450 28663 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Bootstrap starting.
I20260928 16:43:23.303531 28664 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Bootstrap starting.
I20260928 16:43:23.305364 28663 tablet_bootstrap.cc:654] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.306926 28664 tablet_bootstrap.cc:654] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.307507 28663 log.cc:824] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:23.308275 28664 log.cc:824] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:23.310518 28663 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: No bootstrap required, opened a new log
I20260928 16:43:23.310784 28663 ts_tablet_manager.cc:1397] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 16:43:23.311223 28664 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: No bootstrap required, opened a new log
I20260928 16:43:23.311492 28664 ts_tablet_manager.cc:1397] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 16:43:23.312345 28667 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Bootstrap starting.
I20260928 16:43:23.316241 28667 tablet_bootstrap.cc:654] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.317490 28667 log.cc:824] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:23.319655 28664 raft_consensus.cc:348] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.320031 28664 raft_consensus.cc:374] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.320159 28664 raft_consensus.cc:729] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e92562b7034c449c0ef0a04617674b, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.320761 28664 consensus_queue.cc:260] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.321048 28663 raft_consensus.cc:348] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.321590 28663 raft_consensus.cc:374] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.321686 28663 raft_consensus.cc:729] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1254e1b365d5431ba5cf9c80ef544d02, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.321846 28504 heartbeater.cc:503] Master 127.26.101.62:43261 was elected leader, sending a full tablet report...
I20260928 16:43:23.322372 28664 ts_tablet_manager.cc:1428] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260928 16:43:23.322384 28663 consensus_queue.cc:260] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.323532 28644 heartbeater.cc:503] Master 127.26.101.62:43261 was elected leader, sending a full tablet report...
I20260928 16:43:23.324146 28663 ts_tablet_manager.cc:1428] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 16:43:23.327399 28667 tablet_bootstrap.cc:492] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: No bootstrap required, opened a new log
I20260928 16:43:23.327708 28667 ts_tablet_manager.cc:1397] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260928 16:43:23.335356 28667 raft_consensus.cc:348] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.335747 28667 raft_consensus.cc:374] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.335860 28667 raft_consensus.cc:729] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc2fb5c62014dc5970163c212585a7a, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.336416 28667 consensus_queue.cc:260] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.337858 28667 ts_tablet_manager.cc:1428] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260928 16:43:23.404443 28365 tablet.cc:2369] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:23.408555 28505 tablet.cc:2369] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:23.421842 28645 tablet.cc:2369] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:23.444474 28669 raft_consensus.cc:482] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:23.444813 28669 raft_consensus.cc:504] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.446550 28669 leader_election.cc:288] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163), 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193)
I20260928 16:43:23.453809 28670 raft_consensus.cc:482] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:23.454167 28670 raft_consensus.cc:504] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.454658 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.447108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:60603 (local address 127.26.101.3:38163)
0928 16:43:23.447350 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.447357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.447816 (+ 459us) server_negotiation.cc:407] Connection header received
0928 16:43:23.448082 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.448089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.448180 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.448278 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.448952 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.449916 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.452398 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.453610 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.453813 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.453903 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.454247 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.454407 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:43:23.454774 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.448177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:48623 (local address 127.26.101.1:44193)
0928 16:43:23.448475 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.448481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.448638 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:43:23.448814 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.448818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.448896 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.448993 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.449477 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.450356 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.452639 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.453842 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.454040 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.454111 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.454425 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.454553 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 16:43:23.454921 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.447833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44193 (local address 127.26.101.2:48623)
0928 16:43:23.448548 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.448568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.448689 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.449012 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.449015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.449034 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.449338 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.449347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.450502 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.450508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.452499 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.452510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.454086 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.454092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.454260 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.454291 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.454380 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.454518 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1,"thread_start_us":324,"threads_started":1}
I20260928 16:43:23.455394 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.446954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:38163 (local address 127.26.101.2:60603)
0928 16:43:23.447702 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.447719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.447878 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.448326 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.448330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.448363 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.448771 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.448783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.450047 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.450051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.452219 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.452235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.453848 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.453855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.454044 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.454084 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.454911 (+ 827us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.455048 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":432,"threads_started":1}
I20260928 16:43:23.456166 28313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a" is_pre_election: true
I20260928 16:43:23.456336 28670 leader_election.cc:288] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193), c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415)
I20260928 16:43:23.456722 28313 raft_consensus.cc:2454] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 0.
I20260928 16:43:23.457491 28386 leader_election.cc:302] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc2fb5c62014dc5970163c212585a7a, c8e92562b7034c449c0ef0a04617674b; no voters:
I20260928 16:43:23.456597 28593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1254e1b365d5431ba5cf9c80ef544d02" is_pre_election: true
I20260928 16:43:23.458389 28593 raft_consensus.cc:2454] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 0.
I20260928 16:43:23.458503 28669 raft_consensus.cc:2781] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:23.458662 28669 raft_consensus.cc:482] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:23.458760 28669 raft_consensus.cc:3037] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.462013 28669 raft_consensus.cc:504] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.462630 28669 leader_election.cc:288] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 election: Requested vote from peers 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163), 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193)
I20260928 16:43:23.463246 28593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1254e1b365d5431ba5cf9c80ef544d02"
I20260928 16:43:23.463239 28313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a"
I20260928 16:43:23.463471 28593 raft_consensus.cc:3037] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.463460 28313 raft_consensus.cc:3037] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.464118 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.456578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44193 (local address 127.26.101.3:48993)
0928 16:43:23.457340 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.457364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.457487 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.457852 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.457856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.457885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.458097 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.458108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.459610 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.459615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.461630 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.461646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.463330 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.463336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.463545 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.463584 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.463641 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.463797 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260928 16:43:23.464928 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.457067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:48993 (local address 127.26.101.1:44193)
0928 16:43:23.457343 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.457348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.457450 (+ 102us) server_negotiation.cc:407] Connection header received
0928 16:43:23.457621 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.457626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.457701 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.458245 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.458376 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.459434 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.461807 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.463140 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.464369 (+ 1229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.464449 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.464503 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.464641 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 16:43:23.465958 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.457031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:51975 (local address 127.26.101.2:33415)
0928 16:43:23.457223 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.457228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.459690 (+ 2462us) server_negotiation.cc:407] Connection header received
0928 16:43:23.459854 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.459858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.459917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.459991 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:23.460712 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.461445 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.463758 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.464891 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.465023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.465090 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.465583 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.465711 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:43:23.466022 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.457450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33415 (local address 127.26.101.3:51975)
0928 16:43:23.459596 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.459611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.459728 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.460141 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.460145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.460170 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:23.460550 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.460558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.461579 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.461583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.463564 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.463578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.465285 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.465289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.465428 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.465455 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.465532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.465664 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2015,"thread_start_us":750,"threads_started":1}
I20260928 16:43:23.466720 28313 raft_consensus.cc:2454] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 1.
I20260928 16:43:23.466720 28593 raft_consensus.cc:2454] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 1.
I20260928 16:43:23.467288 28386 leader_election.cc:302] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc2fb5c62014dc5970163c212585a7a, c8e92562b7034c449c0ef0a04617674b; no voters:
I20260928 16:43:23.467127 28453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e92562b7034c449c0ef0a04617674b" is_pre_election: true
I20260928 16:43:23.467329 28312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9abbbfb7247cea8e36fa99d2b0fbe" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a" is_pre_election: true
I20260928 16:43:23.467701 28453 raft_consensus.cc:2379] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1254e1b365d5431ba5cf9c80ef544d02 in current term 1: Already voted for candidate c8e92562b7034c449c0ef0a04617674b in this term.
I20260928 16:43:23.467716 28312 raft_consensus.cc:2379] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1254e1b365d5431ba5cf9c80ef544d02 in current term 1: Already voted for candidate c8e92562b7034c449c0ef0a04617674b in this term.
I20260928 16:43:23.468292 28669 raft_consensus.cc:2781] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:23.468695 28527 leader_election.cc:302] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1254e1b365d5431ba5cf9c80ef544d02; no voters: 5dc2fb5c62014dc5970163c212585a7a, c8e92562b7034c449c0ef0a04617674b
I20260928 16:43:23.469162 28670 raft_consensus.cc:2726] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:43:23.469656 28669 raft_consensus.cc:686] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 LEADER]: Becoming Leader. State: Replica: c8e92562b7034c449c0ef0a04617674b, State: Running, Role: LEADER
I20260928 16:43:23.470160 28669 consensus_queue.cc:237] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.475299 28154 catalog_manager.cc:5495] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b reported cstate change: term changed from 0 to 1, leader changed from <none> to c8e92562b7034c449c0ef0a04617674b (127.26.101.2). New cstate: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } health_report { overall_health: HEALTHY } } }
I20260928 16:43:23.528926 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.523242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.26.101.2:33415)
0928 16:43:23.523432 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.523437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.523541 (+ 104us) server_negotiation.cc:407] Connection header received
0928 16:43:23.523900 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.523904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.524002 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.524139 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:23.524471 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.525277 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.526650 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.527171 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.527282 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.527766 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:23.528361 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:23.528472 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.528568 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.528643 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:43:23.537602 28312 raft_consensus.cc:1261] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Refusing update from remote peer c8e92562b7034c449c0ef0a04617674b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:23.537601 28593 raft_consensus.cc:1261] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Refusing update from remote peer c8e92562b7034c449c0ef0a04617674b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:23.538589 28675 consensus_queue.cc:1035] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:23.538973 28669 consensus_queue.cc:1035] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:23.551628 28680 mvcc.cc:205] Tried to move back new op lower bound from 7334354139278888960 to 7334354139017953280. Current Snapshot: MvccSnapshot[applied={T|T < 7334354139278888960}]
I20260928 16:43:23.561185 28679 mvcc.cc:205] Tried to move back new op lower bound from 7334354139278888960 to 7334354139017953280. Current Snapshot: MvccSnapshot[applied={T|T < 7334354139278888960}]
I20260928 16:43:23.578173 28153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:23.578554 28153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:23.583635 28153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Sending DeleteTablet for 3 replicas of tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe
I20260928 16:43:23.585179 28573 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:23 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51946
I20260928 16:43:23.586824 28293 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:23 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:44848
I20260928 16:43:23.587234 28433 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:23 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:35834
I20260928 16:43:23.588178 28688 tablet_replica.cc:330] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: stopping tablet replica
I20260928 16:43:23.588670 28688 raft_consensus.cc:2229] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:23.588624 28153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:43:23.589365 28688 raft_consensus.cc:2258] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:43:23.589392 28153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:23.591686 28688 ts_tablet_manager.cc:1905] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:23.599534 28688 ts_tablet_manager.cc:1918] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:23.599857 28688 log.cc:1196] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/3ff9abbbfb7247cea8e36fa99d2b0fbe
I20260928 16:43:23.600518 28688 ts_tablet_manager.cc:1939] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 1254e1b365d5431ba5cf9c80ef544d02: Deleting consensus metadata
I20260928 16:43:23.600853 28573 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b (DEFAULT_TABLE table=test-table [id=5b3edd0702d648428685e590e0ec6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.601398 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1bad1c12c42f9be7f1f4a6662d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.604025 28292 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b (DEFAULT_TABLE table=test-table [id=5b3edd0702d648428685e590e0ec6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.604655 28292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1bad1c12c42f9be7f1f4a6662d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.605049 28432 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b (DEFAULT_TABLE table=test-table [id=5b3edd0702d648428685e590e0ec6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:23.605638 28432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1bad1c12c42f9be7f1f4a6662d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.603897 28140 catalog_manager.cc:4841] TS 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163): tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (table test-table [id=3161988dd631432b8a7689fd29985b73]) successfully deleted
I20260928 16:43:23.611964 28667 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Bootstrap starting.
I20260928 16:43:23.612987 28663 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Bootstrap starting.
I20260928 16:43:23.613741 28667 tablet_bootstrap.cc:654] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.614632 28663 tablet_bootstrap.cc:654] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.616184 28667 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: No bootstrap required, opened a new log
I20260928 16:43:23.616350 28667 ts_tablet_manager.cc:1397] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 16:43:23.617178 28667 raft_consensus.cc:348] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.617424 28667 raft_consensus.cc:374] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.617549 28667 raft_consensus.cc:729] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc2fb5c62014dc5970163c212585a7a, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.617842 28667 consensus_queue.cc:260] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.618546 28667 ts_tablet_manager.cc:1428] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:43:23.619330 28664 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Bootstrap starting.
I20260928 16:43:23.621179 28664 tablet_bootstrap.cc:654] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.623732 28664 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: No bootstrap required, opened a new log
I20260928 16:43:23.623953 28664 ts_tablet_manager.cc:1397] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 16:43:23.624840 28664 raft_consensus.cc:348] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.625106 28664 raft_consensus.cc:374] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.625243 28664 raft_consensus.cc:729] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e92562b7034c449c0ef0a04617674b, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.625514 28664 consensus_queue.cc:260] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.626850 28664 ts_tablet_manager.cc:1428] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:23.628928 28663 tablet_bootstrap.cc:492] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: No bootstrap required, opened a new log
I20260928 16:43:23.629172 28663 ts_tablet_manager.cc:1397] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260928 16:43:23.630108 28663 raft_consensus.cc:348] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.630434 28663 raft_consensus.cc:374] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.630554 28663 raft_consensus.cc:729] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1254e1b365d5431ba5cf9c80ef544d02, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.630870 28663 consensus_queue.cc:260] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.632149 28663 ts_tablet_manager.cc:1428] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:43:23.634940 28690 tablet_replica.cc:330] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: stopping tablet replica
I20260928 16:43:23.635427 28689 tablet_replica.cc:330] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: stopping tablet replica
I20260928 16:43:23.635936 28689 raft_consensus.cc:2229] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:23.635527 28690 raft_consensus.cc:2229] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:23.636488 28689 raft_consensus.cc:2258] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:23.637166 28690 raft_consensus.cc:2258] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:23.638875 28689 ts_tablet_manager.cc:1905] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:23.639698 28690 ts_tablet_manager.cc:1905] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:23.645957 28689 ts_tablet_manager.cc:1918] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:23.646144 28689 log.cc:1196] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/3ff9abbbfb7247cea8e36fa99d2b0fbe
I20260928 16:43:23.646673 28689 ts_tablet_manager.cc:1939] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P 5dc2fb5c62014dc5970163c212585a7a: Deleting consensus metadata
I20260928 16:43:23.647637 28690 ts_tablet_manager.cc:1918] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:23.647935 28690 log.cc:1196] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/3ff9abbbfb7247cea8e36fa99d2b0fbe
I20260928 16:43:23.648525 28690 ts_tablet_manager.cc:1939] T 3ff9abbbfb7247cea8e36fa99d2b0fbe P c8e92562b7034c449c0ef0a04617674b: Deleting consensus metadata
I20260928 16:43:23.648736 28140 catalog_manager.cc:4841] TS 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193): tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (table test-table [id=3161988dd631432b8a7689fd29985b73]) successfully deleted
I20260928 16:43:23.650180 28141 catalog_manager.cc:4841] TS c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415): tablet 3ff9abbbfb7247cea8e36fa99d2b0fbe (table test-table [id=3161988dd631432b8a7689fd29985b73]) successfully deleted
I20260928 16:43:23.709403 28670 raft_consensus.cc:482] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:23.709620 28670 raft_consensus.cc:504] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.710403 28670 leader_election.cc:288] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193), c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415)
I20260928 16:43:23.710932 28312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1bad1c12c42f9be7f1f4a6662d89b" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a" is_pre_election: true
I20260928 16:43:23.711054 28453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1bad1c12c42f9be7f1f4a6662d89b" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e92562b7034c449c0ef0a04617674b" is_pre_election: true
I20260928 16:43:23.711267 28312 raft_consensus.cc:2454] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1254e1b365d5431ba5cf9c80ef544d02 in term 0.
I20260928 16:43:23.711369 28453 raft_consensus.cc:2454] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1254e1b365d5431ba5cf9c80ef544d02 in term 0.
I20260928 16:43:23.711766 28526 leader_election.cc:302] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1254e1b365d5431ba5cf9c80ef544d02, 5dc2fb5c62014dc5970163c212585a7a; no voters:
I20260928 16:43:23.712069 28670 raft_consensus.cc:2781] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:23.712213 28670 raft_consensus.cc:482] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:23.712318 28670 raft_consensus.cc:3037] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.715500 28670 raft_consensus.cc:504] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.716199 28670 leader_election.cc:288] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193), c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415)
I20260928 16:43:23.716784 28453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1bad1c12c42f9be7f1f4a6662d89b" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e92562b7034c449c0ef0a04617674b"
I20260928 16:43:23.717016 28453 raft_consensus.cc:3037] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.719323 28312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1bad1c12c42f9be7f1f4a6662d89b" candidate_uuid: "1254e1b365d5431ba5cf9c80ef544d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a"
I20260928 16:43:23.719553 28312 raft_consensus.cc:3037] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:23.720415 28453 raft_consensus.cc:2454] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1254e1b365d5431ba5cf9c80ef544d02 in term 1.
I20260928 16:43:23.720896 28527 leader_election.cc:302] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1254e1b365d5431ba5cf9c80ef544d02, c8e92562b7034c449c0ef0a04617674b; no voters:
I20260928 16:43:23.721215 28670 raft_consensus.cc:2781] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:23.722124 28312 raft_consensus.cc:2454] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1254e1b365d5431ba5cf9c80ef544d02 in term 1.
I20260928 16:43:23.722831 28670 raft_consensus.cc:686] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 LEADER]: Becoming Leader. State: Replica: 1254e1b365d5431ba5cf9c80ef544d02, State: Running, Role: LEADER
I20260928 16:43:23.723261 28670 consensus_queue.cc:237] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } }
I20260928 16:43:23.726907 28153 catalog_manager.cc:5495] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "1254e1b365d5431ba5cf9c80ef544d02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } health_report { overall_health: HEALTHY } } }
Sep 28 16:43:23 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613801, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:23.791337 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.777832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:43261 (local address 127.0.0.1:54214)
0928 16:43:23.778491 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:23.778507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:23.778656 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:23.778973 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:23.778977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:23.779744 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:23.779983 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.779992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.781091 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.781096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.782047 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:23.782059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.782763 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.782778 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:23.782965 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.783714 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:23.783732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:23.786007 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:23.788860 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.788873 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.788882 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.789279 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.789611 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:23.789616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:23.789620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:23.789790 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:23.790288 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:23.790296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:23.790477 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 16:43:23.790626 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 16:43:23.790898 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":372,"threads_started":1}
I20260928 16:43:23.791432 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:23.777678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.26.101.62:43261)
0928 16:43:23.778324 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:23.778330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:23.778600 (+ 270us) server_negotiation.cc:407] Connection header received
0928 16:43:23.778771 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:23.778776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:23.778837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:23.778933 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:23.780152 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.780960 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.782178 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:23.782618 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:23.782721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:23.786222 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:23.786227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:23.786230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:23.786260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:23.788731 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.789388 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.789393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.789396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.789512 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:23.789890 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:23.789901 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:23.789903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:23.790167 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:23.790295 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:23.790718 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 16:43:23.791038 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":471,"thread_start_us":302,"threads_started":1}
I20260928 16:43:23.796517 28153 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:54214:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:43:23.797509 28153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:23.805027 28573 tablet_service.cc:1467] Processing CreateTablet for tablet 2879a648b4ed423c9f870d5d71df4320 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=848e4fc29a90467790de65f57fecd7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:23.805024 28293 tablet_service.cc:1467] Processing CreateTablet for tablet 2879a648b4ed423c9f870d5d71df4320 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=848e4fc29a90467790de65f57fecd7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:23.805384 28433 tablet_service.cc:1467] Processing CreateTablet for tablet 2879a648b4ed423c9f870d5d71df4320 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=848e4fc29a90467790de65f57fecd7dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:23.805660 28293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2879a648b4ed423c9f870d5d71df4320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.805660 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2879a648b4ed423c9f870d5d71df4320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.806013 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2879a648b4ed423c9f870d5d71df4320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:23.813186 28663 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: Bootstrap starting.
I20260928 16:43:23.813186 28667 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: Bootstrap starting.
I20260928 16:43:23.814965 28663 tablet_bootstrap.cc:654] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.815986 28667 tablet_bootstrap.cc:654] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.817480 28664 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: Bootstrap starting.
I20260928 16:43:23.819284 28664 tablet_bootstrap.cc:654] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:23.821857 28664 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: No bootstrap required, opened a new log
I20260928 16:43:23.822053 28664 ts_tablet_manager.cc:1397] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 16:43:23.822918 28664 raft_consensus.cc:348] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.823256 28664 raft_consensus.cc:374] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.823341 28664 raft_consensus.cc:729] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e92562b7034c449c0ef0a04617674b, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.823663 28664 consensus_queue.cc:260] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.824091 28685 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:23.824533 28685 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:23.824808 28663 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: No bootstrap required, opened a new log
I20260928 16:43:23.824983 28663 ts_tablet_manager.cc:1397] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260928 16:43:23.825199 28664 ts_tablet_manager.cc:1428] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:23.825755 28663 raft_consensus.cc:348] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.825994 28663 raft_consensus.cc:374] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.826062 28663 raft_consensus.cc:729] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1254e1b365d5431ba5cf9c80ef544d02, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.826633 28663 consensus_queue.cc:260] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.827224 28663 ts_tablet_manager.cc:1428] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 16:43:23.827579 28670 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:23.827814 28670 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:23.828874 28667 tablet_bootstrap.cc:492] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: No bootstrap required, opened a new log
I20260928 16:43:23.829051 28667 ts_tablet_manager.cc:1397] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20260928 16:43:23.829865 28667 raft_consensus.cc:348] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.830103 28667 raft_consensus.cc:374] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:23.830168 28667 raft_consensus.cc:729] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc2fb5c62014dc5970163c212585a7a, State: Initialized, Role: FOLLOWER
I20260928 16:43:23.830538 28667 consensus_queue.cc:260] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:23.831177 28667 ts_tablet_manager.cc:1428] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 16:43:23.831037 28671 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:23.831465 28671 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:24.134706 28685 raft_consensus.cc:482] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:24.134923 28685 raft_consensus.cc:504] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:24.135602 28685 leader_election.cc:288] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163), 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193)
I20260928 16:43:24.136360 28593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2879a648b4ed423c9f870d5d71df4320" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1254e1b365d5431ba5cf9c80ef544d02" is_pre_election: true
I20260928 16:43:24.136353 28312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2879a648b4ed423c9f870d5d71df4320" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a" is_pre_election: true
I20260928 16:43:24.136634 28312 raft_consensus.cc:2454] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 0.
I20260928 16:43:24.136634 28593 raft_consensus.cc:2454] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 0.
I20260928 16:43:24.137082 28386 leader_election.cc:302] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc2fb5c62014dc5970163c212585a7a, c8e92562b7034c449c0ef0a04617674b; no voters:
I20260928 16:43:24.137362 28685 raft_consensus.cc:2781] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:24.137482 28685 raft_consensus.cc:482] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:24.137598 28685 raft_consensus.cc:3037] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:24.139729 28685 raft_consensus.cc:504] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:24.140255 28685 leader_election.cc:288] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 election: Requested vote from peers 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163), 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193)
I20260928 16:43:24.140856 28593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2879a648b4ed423c9f870d5d71df4320" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1254e1b365d5431ba5cf9c80ef544d02"
I20260928 16:43:24.140928 28312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2879a648b4ed423c9f870d5d71df4320" candidate_uuid: "c8e92562b7034c449c0ef0a04617674b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc2fb5c62014dc5970163c212585a7a"
I20260928 16:43:24.141054 28593 raft_consensus.cc:3037] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:24.141139 28312 raft_consensus.cc:3037] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:24.143095 28593 raft_consensus.cc:2454] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 1.
I20260928 16:43:24.143162 28312 raft_consensus.cc:2454] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e92562b7034c449c0ef0a04617674b in term 1.
I20260928 16:43:24.143532 28386 leader_election.cc:302] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1254e1b365d5431ba5cf9c80ef544d02, c8e92562b7034c449c0ef0a04617674b; no voters:
I20260928 16:43:24.143837 28685 raft_consensus.cc:2781] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:24.144196 28685 raft_consensus.cc:686] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [term 1 LEADER]: Becoming Leader. State: Replica: c8e92562b7034c449c0ef0a04617674b, State: Running, Role: LEADER
I20260928 16:43:24.144460 28685 consensus_queue.cc:237] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } }
I20260928 16:43:24.147284 28706 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: TxnStatusTablet state changed. Reason: New leader c8e92562b7034c449c0ef0a04617674b. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.147527 28706 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:24.148875 28708 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:43:24.148715 28154 catalog_manager.cc:5495] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b reported cstate change: term changed from 0 to 1, leader changed from <none> to c8e92562b7034c449c0ef0a04617674b (127.26.101.2). New cstate: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } health_report { overall_health: HEALTHY } } }
I20260928 16:43:24.149202 28708 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:43:24.200634 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.189710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33415 (local address 127.0.0.1:35852)
0928 16:43:24.189979 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.189999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:24.190147 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:24.195531 (+ 5384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:24.195536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:24.195550 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:24.195760 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.195773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.197054 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.197059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.198110 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.198124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.199109 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.199116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.199306 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.199395 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.200021 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.200068 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 16:43:24.200131 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:43:24.200309 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 16:43:24.200639 28670 consensus_queue.cc:1035] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:24.201452 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.190058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.26.101.2:33415)
0928 16:43:24.195131 (+ 5073us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.195138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.195161 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:43:24.195231 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.195235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.195331 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.195485 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:24.195944 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.196898 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.198260 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.198881 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.199621 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.199765 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.199894 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.200928 (+ 1034us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.200981 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.201064 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4906,"thread_start_us":4752,"threads_started":1}
I20260928 16:43:24.208307 28312 raft_consensus.cc:1261] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Refusing update from remote peer c8e92562b7034c449c0ef0a04617674b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:24.209074 28706 consensus_queue.cc:1035] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:24.211169 28593 raft_consensus.cc:1261] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Refusing update from remote peer c8e92562b7034c449c0ef0a04617674b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:24.211302 28670 consensus_queue.cc:1035] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:24.213155 28685 consensus_queue.cc:1035] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:24.218713 28670 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: TxnStatusTablet state changed. Reason: New leader c8e92562b7034c449c0ef0a04617674b. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.219025 28670 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:24.219647 28713 mvcc.cc:205] Tried to move back new op lower bound from 7334354142031114240 to 7334354141777981440. Current Snapshot: MvccSnapshot[applied={T|T < 7334354142031114240}]
I20260928 16:43:24.227551 28685 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.227917 28685 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:24.231288 28694 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.231637 28694 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 1254e1b365d5431ba5cf9c80ef544d02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:24.236399 28671 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: TxnStatusTablet state changed. Reason: New leader c8e92562b7034c449c0ef0a04617674b. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.236687 28671 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:24.239853 28706 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.240141 28706 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P c8e92562b7034c449c0ef0a04617674b: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:24.246093 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.236511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.26.101.3:38163)
0928 16:43:24.237978 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.237983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.238002 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:43:24.238106 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.238112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.238212 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.238364 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:24.239114 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.240333 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.242946 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.243714 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.243819 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.244543 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.245375 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.245527 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.245655 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.245753 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":1221,"threads_started":1}
I20260928 16:43:24.250080 28671 tablet_replica.cc:439] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e92562b7034c449c0ef0a04617674b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1254e1b365d5431ba5cf9c80ef544d02" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38163 } } peers { permanent_uuid: "5dc2fb5c62014dc5970163c212585a7a" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44193 } } peers { permanent_uuid: "c8e92562b7034c449c0ef0a04617674b" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33415 } } }
I20260928 16:43:24.250365 28671 tablet_replica.cc:442] T 2879a648b4ed423c9f870d5d71df4320 P 5dc2fb5c62014dc5970163c212585a7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:24.263553 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.257487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33415 (local address 127.26.101.3:37459)
0928 16:43:24.258090 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.258108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:24.258254 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:24.258588 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:24.258592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:24.258601 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:24.258840 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.258847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.260315 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.260319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.261380 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.261407 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.262379 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.262384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.262536 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.262598 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.263023 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.263039 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:43:24.263124 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:43:24.263237 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260928 16:43:24.263566 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.257496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:37459 (local address 127.26.101.2:33415)
0928 16:43:24.257697 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.257703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.258207 (+ 504us) server_negotiation.cc:407] Connection header received
0928 16:43:24.258384 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.258390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.258478 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.258574 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:24.259000 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.260187 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.261541 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.262196 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.262326 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.262770 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.262929 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.263013 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.263181 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.263272 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:43:24.307286 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.299596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:53855 (local address 127.26.101.3:38163)
0928 16:43:24.299777 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.299783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.300540 (+ 757us) server_negotiation.cc:407] Connection header received
0928 16:43:24.300860 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.300866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.300950 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.301010 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:24.301872 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.303000 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.305205 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.305857 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.305954 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.306642 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.306747 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.306815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.306964 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.307033 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:43:24.307600 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.299596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:38163 (local address 127.26.101.2:53855)
0928 16:43:24.300439 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:24.300459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:24.300668 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:24.301215 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:24.301220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:24.301233 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:24.301677 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.301689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.303225 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.303231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.305023 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:24.305041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.306157 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.306164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:24.306374 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.306454 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:24.306848 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:24.306860 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:43:24.306941 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:43:24.307146 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":382,"threads_started":1}
I20260928 16:43:24.351336 28154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:24.351644 28154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:24.355370 28154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58f60977939e45ff8139dbda7a0b453b: Sending DeleteTablet for 3 replicas of tablet e0c1bad1c12c42f9be7f1f4a6662d89b
I20260928 16:43:24.356469 28293 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:24 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:44848
I20260928 16:43:24.356777 28573 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:24 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51946
I20260928 16:43:24.358923 28433 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c1bad1c12c42f9be7f1f4a6662d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:24 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:35834
I20260928 16:43:24.381807 28737 tablet_replica.cc:330] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: stopping tablet replica
I20260928 16:43:24.382229 28737 raft_consensus.cc:2229] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:24.382874 28739 tablet_replica.cc:330] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: stopping tablet replica
I20260928 16:43:24.383594 28737 raft_consensus.cc:2258] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:24.383932 28739 raft_consensus.cc:2229] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:24.384254 28739 raft_consensus.cc:2258] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:24.386482 28736 tablet_replica.cc:330] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: stopping tablet replica
I20260928 16:43:24.386691 28736 raft_consensus.cc:2229] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:24.386971 28736 raft_consensus.cc:2258] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:24.386499 28739 ts_tablet_manager.cc:1905] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:24.387007 28737 ts_tablet_manager.cc:1905] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:24.390841 28736 ts_tablet_manager.cc:1905] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:24.399531 28736 ts_tablet_manager.cc:1918] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:24.399780 28736 log.cc:1196] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/e0c1bad1c12c42f9be7f1f4a6662d89b
I20260928 16:43:24.400379 28736 ts_tablet_manager.cc:1939] T e0c1bad1c12c42f9be7f1f4a6662d89b P 5dc2fb5c62014dc5970163c212585a7a: Deleting consensus metadata
I20260928 16:43:24.401772 28140 catalog_manager.cc:4841] TS 5dc2fb5c62014dc5970163c212585a7a (127.26.101.1:44193): tablet e0c1bad1c12c42f9be7f1f4a6662d89b (table test-table [id=5b3edd0702d648428685e590e0ec6ec8]) successfully deleted
I20260928 16:43:24.403664 28739 ts_tablet_manager.cc:1918] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:24.403924 28739 log.cc:1196] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/e0c1bad1c12c42f9be7f1f4a6662d89b
I20260928 16:43:24.404486 28739 ts_tablet_manager.cc:1939] T e0c1bad1c12c42f9be7f1f4a6662d89b P c8e92562b7034c449c0ef0a04617674b: Deleting consensus metadata
I20260928 16:43:24.405797 28141 catalog_manager.cc:4841] TS c8e92562b7034c449c0ef0a04617674b (127.26.101.2:33415): tablet e0c1bad1c12c42f9be7f1f4a6662d89b (table test-table [id=5b3edd0702d648428685e590e0ec6ec8]) successfully deleted
I20260928 16:43:24.407683 28737 ts_tablet_manager.cc:1918] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:24.407840 28737 log.cc:1196] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/e0c1bad1c12c42f9be7f1f4a6662d89b
I20260928 16:43:24.408331 28737 ts_tablet_manager.cc:1939] T e0c1bad1c12c42f9be7f1f4a6662d89b P 1254e1b365d5431ba5cf9c80ef544d02: Deleting consensus metadata
I20260928 16:43:24.409673 28140 catalog_manager.cc:4841] TS 1254e1b365d5431ba5cf9c80ef544d02 (127.26.101.3:38163): tablet e0c1bad1c12c42f9be7f1f4a6662d89b (table test-table [id=5b3edd0702d648428685e590e0ec6ec8]) successfully deleted
I20260928 16:43:24.417650 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.366773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.26.101.1:44193)
0928 16:43:24.381206 (+ 14433us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.381213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.382347 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 16:43:24.382490 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.382495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.382592 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.382707 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:24.384926 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.386023 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.389020 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.389686 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.391040 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.409589 (+ 18549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:24.409611 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:24.409632 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:24.409684 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:24.414030 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.415142 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.415153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.415157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.415363 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.415821 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.415826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.415830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.416305 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:24.416638 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.416974 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.417231 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14269,"thread_start_us":14443,"threads_started":1}
W20260928 16:43:24.418491 28359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44852
I20260928 16:43:24.452720 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.422314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.26.101.62:43261)
0928 16:43:24.435605 (+ 13291us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.435612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.435641 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:43:24.435717 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.435722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.435813 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.435971 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:24.437950 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.439216 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.441164 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.441866 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.442019 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.445455 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:24.445463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:24.445468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:24.445518 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:24.449379 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.450279 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.450285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.450289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.450456 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.450858 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.450863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.450866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.451275 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:24.451533 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.451874 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.452138 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13133,"thread_start_us":12946,"threads_started":1}
W20260928 16:43:24.453413 28154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54224
I20260928 16:43:24.455694 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28226
I20260928 16:43:24.478612 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.458225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.26.101.1:44193)
0928 16:43:24.458459 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.458466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.462886 (+ 4420us) server_negotiation.cc:407] Connection header received
0928 16:43:24.463102 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.463109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.463201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.463324 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:24.464870 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.465938 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.467826 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.468502 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.468642 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.471656 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:24.471662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:24.471667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:24.471716 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:24.475407 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.476324 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.476331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.476335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.476507 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.476983 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.476989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.476993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.477356 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:24.477632 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.477984 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.478236 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 16:43:24.479323 28359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44866
W20260928 16:43:24.480000 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28226: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:24.480206 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28226
I20260928 16:43:24.505323 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28368
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:24.526441 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.506198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.26.101.2:33415)
0928 16:43:24.506456 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.506464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.506763 (+ 299us) server_negotiation.cc:407] Connection header received
0928 16:43:24.506978 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.506984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.507129 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.507232 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:24.509012 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.510219 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.512100 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.512841 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.512999 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.518637 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:24.518665 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:24.518692 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:24.518750 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:24.523006 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.523944 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.523954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.523958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.524155 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.524604 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.524609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.524612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.525129 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:24.525483 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.525779 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.526029 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 16:43:24.527419 28499 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35856
W20260928 16:43:24.528194 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:24.528419 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28368
I20260928 16:43:24.551729 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28508
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:24.572408 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:24.552566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.26.101.3:38163)
0928 16:43:24.552756 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:24.552762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:24.552880 (+ 118us) server_negotiation.cc:407] Connection header received
0928 16:43:24.553067 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:24.553073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:24.553173 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:24.553287 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:24.554989 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.556312 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.558167 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:24.558916 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:24.559052 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:24.564806 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:24.564828 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:24.564850 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:24.564914 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:24.569070 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.570061 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.570071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.570075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.570244 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:24.570648 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:24.570653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:24.570657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:24.571156 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:24.571475 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:24.571765 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:43:24.572021 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 16:43:24.573253 28639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51966
W20260928 16:43:24.574003 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28508: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:24.574150 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28508
I20260928 16:43:24.591396 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28122
I20260928 16:43:24.592538 28195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:24.708056 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28122
2026-09-28T16:43:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4758 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28749](info): set up 2 sockets
Sep 28 16:43:24 dist-test-slave-t0tw krb5kdc[28749](info): commencing operation
krb5kdc: starting...
W20260928 16:43:25.730876 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:25 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:25.918881 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45339
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:26.243611 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:26.243949 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:26.244074 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:26.251533 28763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:26.251668 28763 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:43:26.251751 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:26.251838 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:26.251905 28763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:26.251991 28763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:26.262136 28763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:26.263552 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:26.264966 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:26.277062 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:26.277029 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:26.277396 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:26.277998 28763 server_base.cc:1034] running on GCE node
I20260928 16:43:26.278825 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:26.280431 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:26.281684 28763 hybrid_clock.cc:648] HybridClock initialized: now 1790613806281569 us; error 58 us; skew 500 ppm
I20260928 16:43:26.282027 28763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:26 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613806, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:26.290988 28763 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:26.292623 28763 webserver.cc:460] Webserver started at http://127.26.101.62:46419/ using document root <none> and password file <none>
I20260928 16:43:26.293365 28763 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:26.293479 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:26.293797 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:26.296839 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "54ed1ff701af429eaa55159fab546208"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "dee1e8985c6f38829ad0bcc55b192976"
server_key_iv: "c43e2ebaf1ef382674ad8c78bb8a6269"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.297588 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "54ed1ff701af429eaa55159fab546208"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "dee1e8985c6f38829ad0bcc55b192976"
server_key_iv: "c43e2ebaf1ef382674ad8c78bb8a6269"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.302771 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:43:26.306196 28778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:26.307585 28763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:26.307865 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "54ed1ff701af429eaa55159fab546208"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "dee1e8985c6f38829ad0bcc55b192976"
server_key_iv: "c43e2ebaf1ef382674ad8c78bb8a6269"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.308079 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:26.326354 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:26.330389 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:26.330641 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:26.351336 28763 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45339
I20260928 16:43:26.351356 28839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45339 every 8 connection(s)
I20260928 16:43:26.352722 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:26.356657 28840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:26.356743 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28763
I20260928 16:43:26.356951 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:26.357789 27028 external_mini_cluster.cc:1428] Setting key f4cbc2b2764512a8b0fa96ef7133035c
I20260928 16:43:26.370478 28840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Bootstrap starting.
Sep 28 16:43:26 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:26.374235 28840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:26.375445 28840 log.cc:824] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:26.378618 28840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: No bootstrap required, opened a new log
I20260928 16:43:26.381317 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.359871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.26.101.62:45339)
0928 16:43:26.360689 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:26.360702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:26.360744 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:26.362816 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:26.362840 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:26.363300 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:26.363634 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:26.365416 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.366717 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.368419 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.369083 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.369214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:26.373865 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:26.373881 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:26.373896 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:26.373933 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:26.377244 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.378020 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.378025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.378031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.378194 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.378583 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.378586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.378588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.379252 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:26.379464 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:26.379910 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 16:43:26.380252 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":387,"threads_started":1}
I20260928 16:43:26.386430 28840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } }
I20260928 16:43:26.386773 28840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:26.386864 28840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ed1ff701af429eaa55159fab546208, State: Initialized, Role: FOLLOWER
I20260928 16:43:26.387440 28840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } }
I20260928 16:43:26.387666 28840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:26.387799 28840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:26.387943 28840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:26.390066 28840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } }
I20260928 16:43:26.390542 28840 leader_election.cc:302] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54ed1ff701af429eaa55159fab546208; no voters:
I20260928 16:43:26.391111 28840 leader_election.cc:288] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:26.391275 28845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:26.391957 28845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [term 1 LEADER]: Becoming Leader. State: Replica: 54ed1ff701af429eaa55159fab546208, State: Running, Role: LEADER
I20260928 16:43:26.392532 28845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } }
I20260928 16:43:26.393065 28840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:26.395445 28846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54ed1ff701af429eaa55159fab546208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } } }
I20260928 16:43:26.395421 28847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54ed1ff701af429eaa55159fab546208. Latest consensus state: current_term: 1 leader_uuid: "54ed1ff701af429eaa55159fab546208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ed1ff701af429eaa55159fab546208" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45339 } } }
I20260928 16:43:26.395893 28846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:26.395895 28847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:26.397740 28850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:26.402452 28850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:26.411681 28850 catalog_manager.cc:1347] Generated new cluster ID: cfa508c3976b413c9d53a00c65418cc6
I20260928 16:43:26.411798 28850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:26.423638 28850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:26.425661 28850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:26.436040 28850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Generated new TSK 0
I20260928 16:43:26.437150 28850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:26.496622 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45339
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:26.817685 28866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:26.818017 28866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:26.818157 28866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:26.825666 28866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:26.825811 28866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:26.825941 28866 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:43:26.826015 28866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:26.826120 28866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:26.836107 28866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:26.837479 28866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:26.838915 28866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:26.852674 28871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:26.852572 28872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:26.852632 28874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:26.853698 28866 server_base.cc:1034] running on GCE node
I20260928 16:43:26.854408 28866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:26.855466 28866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:26.856675 28866 hybrid_clock.cc:648] HybridClock initialized: now 1790613806856622 us; error 51 us; skew 500 ppm
I20260928 16:43:26.857002 28866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:26 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613806, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:26.860224 28866 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:26.861776 28866 webserver.cc:460] Webserver started at http://127.26.101.1:39715/ using document root <none> and password file <none>
I20260928 16:43:26.862470 28866 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:26.862591 28866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:26.862903 28866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:26.865944 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "5add1b4b69004a26b71d68b631804fc8"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "4205500decb5ecf299a2c6127984348d"
server_key_iv: "d1e0465956bc39228012631f1ccf3f3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.866683 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "5add1b4b69004a26b71d68b631804fc8"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "4205500decb5ecf299a2c6127984348d"
server_key_iv: "d1e0465956bc39228012631f1ccf3f3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.871937 28866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:43:26.875485 28881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:26.877019 28866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 16:43:26.877184 28866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "5add1b4b69004a26b71d68b631804fc8"
format_stamp: "Formatted at 2026-09-28 16:43:26 on dist-test-slave-t0tw"
server_key: "4205500decb5ecf299a2c6127984348d"
server_key_iv: "d1e0465956bc39228012631f1ccf3f3e"
server_key_version: "encryptionkey@0"
I20260928 16:43:26.877385 28866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:26.898988 28866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:26.903606 28866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:26.903892 28866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:26.906713 28866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:26.906824 28866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:26.906937 28866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:26.907011 28866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:26 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613806, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:26.931778 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.907328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:40613 (local address 127.26.101.62:45339)
0928 16:43:26.908148 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:26.908155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:26.909273 (+ 1118us) server_negotiation.cc:407] Connection header received
0928 16:43:26.911494 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:26.911502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:26.911602 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:26.911690 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:26.914353 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.915382 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.917719 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.918318 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.918453 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:26.923417 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:26.923423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:26.923426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:26.923465 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:26.926948 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.928098 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.928102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.928105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.928265 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.928961 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.928966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.928970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.929365 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:26.929515 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:26.931294 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0928 16:43:26.931504 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":589,"threads_started":1}
I20260928 16:43:26.933013 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.908130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.1:40613)
0928 16:43:26.909072 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:26.909123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:26.911109 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:26.912323 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:26.912336 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:26.913181 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:26.914129 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:26.914155 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.915544 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.915550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:26.917511 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:26.917535 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.918799 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.918804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:26.919143 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:26.919998 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:26.920044 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:26.923148 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:26.927165 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:26.927178 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:26.927203 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:26.927895 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:26.928413 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:26.928418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:26.928422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:26.928764 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:26.929602 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:26.929612 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:26.930237 (+ 625us) client_negotiation.cc:769] Sending connection context
0928 16:43:26.930634 (+ 397us) client_negotiation.cc:240] Negotiation successful
0928 16:43:26.931371 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":373,"threads_started":1}
I20260928 16:43:26.953894 28866 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:34331
I20260928 16:43:26.953927 29002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:34331 every 8 connection(s)
I20260928 16:43:26.955262 28866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:26.956256 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 28866
I20260928 16:43:26.956457 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:26.957255 27028 external_mini_cluster.cc:1428] Setting key 682f7a27c69fc6d8b388ec3853ae1ea7
I20260928 16:43:26.958590 29003 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45339
I20260928 16:43:26.958881 29003 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:26.959925 29003 heartbeater.cc:511] Master 127.26.101.62:45339 requested a full tablet report, sending...
I20260928 16:43:26.960606 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.938948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.1:55097)
0928 16:43:26.947709 (+ 8761us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:26.947750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:26.947926 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:26.948269 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:26.948274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:26.948771 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:26.949082 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:26.949092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.950126 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.950130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:26.951220 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:26.951235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.952098 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.952106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:26.952273 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:26.952899 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:26.952925 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:26.953842 (+ 917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:26.957183 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:26.957190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:26.957195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:26.957783 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:26.958297 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:26.958302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:26.958307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:26.958476 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:26.959301 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:26.959307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:26.959537 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 16:43:26.959691 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:43:26.960084 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8581,"thread_start_us":8371,"threads_started":1}
I20260928 16:43:26.961613 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:26.938822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55097 (local address 127.26.101.62:45339)
0928 16:43:26.939006 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:26.939013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:26.947894 (+ 8881us) server_negotiation.cc:407] Connection header received
0928 16:43:26.948054 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:26.948059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:26.948136 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:26.948220 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:26.949252 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.949977 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.951365 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:26.951905 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:26.952020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:26.954033 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:26.954039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:26.954043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:26.954085 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:26.957038 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.957963 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.957970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.957974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.958154 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:26.958610 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:26.958615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:26.958619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:26.959017 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:26.959246 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:26.960967 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0928 16:43:26.961237 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:43:26.962841 28794 ts_manager.cc:194] Registered new tserver with Master: 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331)
I20260928 16:43:26.964879 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:40613
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:27.001999 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45339
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:27.338907 29008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:27.339291 29008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:27.339447 29008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:27.347118 29008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:27.347295 29008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:27.347437 29008 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:43:27.347505 29008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:27.347612 29008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:27.362053 29008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:27.363716 29008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:27.365494 29008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:27.380298 29014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:27.380348 29013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:27.380264 29016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:27.381089 29008 server_base.cc:1034] running on GCE node
I20260928 16:43:27.381742 29008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:27.382725 29008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:27.383941 29008 hybrid_clock.cc:648] HybridClock initialized: now 1790613807383897 us; error 34 us; skew 500 ppm
I20260928 16:43:27.384235 29008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:27 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613807, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:27.387463 29008 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:27.389016 29008 webserver.cc:460] Webserver started at http://127.26.101.2:32941/ using document root <none> and password file <none>
I20260928 16:43:27.389770 29008 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:27.389890 29008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:27.390202 29008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:27.393340 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "f774b42c344a48698388dca00aecc222"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "f63fa43b27bfc15aa0157765ef9521eb"
server_key_iv: "b521c08e347400e7e782dc32a1e22488"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.394119 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "f774b42c344a48698388dca00aecc222"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "f63fa43b27bfc15aa0157765ef9521eb"
server_key_iv: "b521c08e347400e7e782dc32a1e22488"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.399732 29008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:43:27.403575 29023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:27.405064 29008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:27.405237 29008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "f774b42c344a48698388dca00aecc222"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "f63fa43b27bfc15aa0157765ef9521eb"
server_key_iv: "b521c08e347400e7e782dc32a1e22488"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.405448 29008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:27.428294 29008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:27.432843 29008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:27.433130 29008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:27.436277 29008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:27.436414 29008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:27.436553 29008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:27.436661 29008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:27 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613807, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:27.461882 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.436784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:41021 (local address 127.26.101.62:45339)
0928 16:43:27.436966 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:27.436974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:27.438736 (+ 1762us) server_negotiation.cc:407] Connection header received
0928 16:43:27.441139 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:27.441146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:27.441244 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:27.441343 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:27.443723 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.444668 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.447882 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.448618 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.448764 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.453915 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:27.453923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:27.453928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:27.453976 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:27.457601 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.458756 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.458762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.458766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.458949 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.459592 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.459598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.459601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.459984 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:27.460133 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:27.461249 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 16:43:27.461512 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:43:27.463022 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.437603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.2:41021)
0928 16:43:27.438528 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.438575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:27.440800 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:27.441660 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:27.441673 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:27.442525 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:27.443516 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.443540 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.444891 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.444898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.447583 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.447614 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.449012 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.449025 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.449283 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.450183 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:27.450230 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:27.453583 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:27.457835 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.457851 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.457877 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.458550 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.459060 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.459069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.459096 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.459422 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.460206 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:27.460215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:27.460795 (+ 580us) client_negotiation.cc:769] Sending connection context
0928 16:43:27.461130 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 16:43:27.461727 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":401,"threads_started":1}
I20260928 16:43:27.483404 29008 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:46165
I20260928 16:43:27.483459 29143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:46165 every 8 connection(s)
I20260928 16:43:27.484853 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.469830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:54787 (local address 127.26.101.62:45339)
0928 16:43:27.470054 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:27.470062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:27.470602 (+ 540us) server_negotiation.cc:407] Connection header received
0928 16:43:27.470755 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:27.470761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:27.470848 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:27.470973 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:27.472006 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.473109 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.475140 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.475733 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.475878 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.478553 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:27.478560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:27.478565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:27.478609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:27.481772 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.482741 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.482746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.482748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.482896 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.483317 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.483329 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.483333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.483751 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:27.483888 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:27.484438 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 16:43:27.484622 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:43:27.485107 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.469859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.2:54787)
0928 16:43:27.470498 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.470517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:27.470645 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:27.471053 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:27.471058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:27.471562 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:27.471859 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.471869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.473316 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.473322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.474863 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.474885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.475943 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.475951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.476174 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.476937 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:27.476968 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:27.478302 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:27.481931 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.481938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.481943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.482597 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.482998 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.483003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.483007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.483196 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.483931 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:27.483938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:27.484162 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 16:43:27.484331 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:43:27.484684 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":335,"threads_started":1}
I20260928 16:43:27.485159 29008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:27.487403 29144 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45339
I20260928 16:43:27.487648 29144 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:27.488368 29144 heartbeater.cc:511] Master 127.26.101.62:45339 requested a full tablet report, sending...
I20260928 16:43:27.489794 28794 ts_manager.cc:194] Registered new tserver with Master: f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:27.490614 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:41021
I20260928 16:43:27.491751 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29008
I20260928 16:43:27.491922 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:27.492365 27028 external_mini_cluster.cc:1428] Setting key dc158e110d95eb708a3f5d4fc5bf0bc1
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:27.534708 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45339
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:27.863652 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:27.863968 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:27.864099 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:27.871539 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:27.871660 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:27.871791 29148 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:43:27.871853 29148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:43:27.871958 29148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:27.881979 29148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:27.883383 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:27.884791 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:27.897707 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:27.897486 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:27.898088 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:27.899151 29148 server_base.cc:1034] running on GCE node
I20260928 16:43:27.899855 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:27.900839 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:27.902077 29148 hybrid_clock.cc:648] HybridClock initialized: now 1790613807902021 us; error 40 us; skew 500 ppm
I20260928 16:43:27.902443 29148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:27 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613807, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:27.905717 29148 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:27.907266 29148 webserver.cc:460] Webserver started at http://127.26.101.3:43385/ using document root <none> and password file <none>
I20260928 16:43:27.907974 29148 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:27.908087 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:27.908363 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:27.911756 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "851722a3ca344203a371b0da8b397480"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "edb136a5d37efc056e54d43d17e5b969"
server_key_iv: "838add762eda0c803b5d7930db003848"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.912580 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "851722a3ca344203a371b0da8b397480"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "edb136a5d37efc056e54d43d17e5b969"
server_key_iv: "838add762eda0c803b5d7930db003848"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.918324 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 16:43:27.922158 29163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:27.923895 29148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:27.924072 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "851722a3ca344203a371b0da8b397480"
format_stamp: "Formatted at 2026-09-28 16:43:27 on dist-test-slave-t0tw"
server_key: "edb136a5d37efc056e54d43d17e5b969"
server_key_iv: "838add762eda0c803b5d7930db003848"
server_key_version: "encryptionkey@0"
I20260928 16:43:27.924291 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:27.937801 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:27.942464 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:27.942765 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:27.945793 29148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:27.945968 29148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:27.946097 29148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:27.946187 29148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:43:27 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613807, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:27.968299 29003 heartbeater.cc:503] Master 127.26.101.62:45339 was elected leader, sending a full tablet report...
I20260928 16:43:27.974617 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.946674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:51913 (local address 127.26.101.62:45339)
0928 16:43:27.946889 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:27.946895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:27.948784 (+ 1889us) server_negotiation.cc:407] Connection header received
0928 16:43:27.950907 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:27.950914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:27.951008 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:27.951177 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:27.953622 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.954738 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.959736 (+ 4998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.960360 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.960507 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.966123 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:27.966153 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:27.966158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:27.966220 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:27.970152 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.971704 (+ 1552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.971711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.971715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.971877 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.972530 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.972534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.972538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.972854 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:27.973020 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:27.974141 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0928 16:43:27.974347 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:43:27.976563 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.947472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.3:51913)
0928 16:43:27.948484 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.948540 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:27.950547 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:27.951464 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:27.951481 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:27.952348 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:27.953365 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.953397 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.954956 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.954962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.959518 (+ 4556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.959536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.960723 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.960734 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.961067 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.962022 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:27.962059 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:27.965803 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:27.970400 (+ 4597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.970417 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.970442 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.971487 (+ 1045us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.972019 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.972025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.972029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.972396 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.973026 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:27.973037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:27.973621 (+ 584us) client_negotiation.cc:769] Sending connection context
0928 16:43:27.973975 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 16:43:27.974652 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":450,"threads_started":1}
I20260928 16:43:27.997372 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.983820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:57809 (local address 127.26.101.62:45339)
0928 16:43:27.983985 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:27.983990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:27.984589 (+ 599us) server_negotiation.cc:407] Connection header received
0928 16:43:27.984765 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:27.984769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:27.984859 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:27.984961 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:27.986125 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.986817 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.988651 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.989082 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.989200 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.991204 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:27.991209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:27.991212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:27.991248 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:27.994240 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.994980 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.994986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.994990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.995180 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:27.995749 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:27.995754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:27.995758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:27.996159 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:27.996302 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:27.996741 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 16:43:27.996983 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:43:27.997429 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:27.983894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.26.101.3:57809)
0928 16:43:27.984460 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:27.984489 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:27.984640 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:27.985133 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:27.985136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:27.985671 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:27.985966 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.985977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.986970 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.986974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.988478 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:27.988497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:27.989278 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:27.989284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:27.989449 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:27.990183 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:27.990200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:27.990972 (+ 772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:27.994400 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.994407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.994412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.994852 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.995369 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:27.995375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:27.995379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:27.995600 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:27.996289 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:27.996294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:27.996484 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 16:43:27.996610 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:43:27.996984 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":302,"threads_started":1}
I20260928 16:43:28.007284 29148 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:43165
I20260928 16:43:28.007306 29283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:43165 every 8 connection(s)
I20260928 16:43:28.008924 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:28.011940 29284 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45339
I20260928 16:43:28.012183 29284 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:28.013069 29284 heartbeater.cc:511] Master 127.26.101.62:45339 requested a full tablet report, sending...
I20260928 16:43:28.013631 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29148
I20260928 16:43:28.013842 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:28.014295 27028 external_mini_cluster.cc:1428] Setting key c79b1c8ff954d62f447efe173dcf9343
I20260928 16:43:28.015661 28794 ts_manager.cc:194] Registered new tserver with Master: 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165)
I20260928 16:43:28.016577 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:51913
I20260928 16:43:28.017913 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:43:28 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:43:28 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:28.065937 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.051065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.26.101.62:45339)
0928 16:43:28.051311 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.051318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.051590 (+ 272us) server_negotiation.cc:407] Connection header received
0928 16:43:28.051774 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.051778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.051847 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.051943 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:28.053348 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.054091 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.055441 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.055856 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.055990 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.059893 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:28.059899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:28.059903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:28.059945 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:28.062900 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:28.063823 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:28.063827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:28.063830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:28.063978 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:28.064367 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:28.064370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:28.064372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:28.064678 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:28.064930 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.065302 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.065570 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:43:28.070218 28795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:28.077397 28795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:28.111304 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.090549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.0.0.1:50510)
0928 16:43:28.098993 (+ 8444us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.099711 (+ 718us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.099961 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.103580 (+ 3619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.103591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.103631 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.104073 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.104089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.106170 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.106176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.108293 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.108307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.110203 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.110209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.110430 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.110499 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.110618 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.110828 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8154,"mutex_wait_us":3,"thread_start_us":13359,"threads_started":1}
I20260928 16:43:28.111855 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.090872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.26.101.3:43165)
0928 16:43:28.102375 (+ 11503us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.102386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.102426 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:43:28.102549 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.102560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.102891 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.103146 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.104338 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.106009 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.108480 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.109655 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.109788 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.110059 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.111260 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.111467 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11295,"thread_start_us":11104,"threads_started":1}
I20260928 16:43:28.112725 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.090742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:34331 (local address 127.0.0.1:55644)
0928 16:43:28.102797 (+ 12055us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.102822 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.102983 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.104940 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.104944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.104964 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.105516 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.105532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.107444 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.107450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.109589 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.109599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.111721 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.111728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.111931 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.112067 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.112141 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.112290 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11879,"mutex_wait_us":81,"thread_start_us":12271,"threads_started":1}
I20260928 16:43:28.112730 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.095001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.26.101.2:46165)
0928 16:43:28.103984 (+ 8983us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.103995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.104032 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:43:28.104315 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.104324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.104628 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.104852 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.105689 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.107019 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.109751 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.111284 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.111826 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.112162 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.112246 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.112431 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8791,"thread_start_us":8652,"threads_started":1}
I20260928 16:43:28.113323 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.090985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.26.101.1:34331)
0928 16:43:28.104441 (+ 13456us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.104450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.104474 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:43:28.104561 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.104567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.104736 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.105222 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.105736 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.107235 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.109753 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.111284 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.112514 (+ 1230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.112727 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.112809 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.113063 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13308,"thread_start_us":13201,"threads_started":1}
I20260928 16:43:28.113654 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.094418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:46165 (local address 127.0.0.1:51706)
0928 16:43:28.103301 (+ 8883us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.103319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.104079 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.104877 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.104883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.104913 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.105450 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.105464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.107173 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.107179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.109535 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.109554 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.111663 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.111670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.111993 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.112030 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.112115 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.112260 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8764,"mutex_wait_us":112,"thread_start_us":4138,"threads_started":1}
I20260928 16:43:28.116251 29213 tablet_service.cc:1467] Processing CreateTablet for tablet dc47764cc9044ff095f74d97abaa437d (DEFAULT_TABLE table=test-table [id=01a84d73ef1c428ca7cd94373ce1a744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.117628 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc47764cc9044ff095f74d97abaa437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.117244 28933 tablet_service.cc:1467] Processing CreateTablet for tablet dc47764cc9044ff095f74d97abaa437d (DEFAULT_TABLE table=test-table [id=01a84d73ef1c428ca7cd94373ce1a744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.117233 29073 tablet_service.cc:1467] Processing CreateTablet for tablet dc47764cc9044ff095f74d97abaa437d (DEFAULT_TABLE table=test-table [id=01a84d73ef1c428ca7cd94373ce1a744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.118747 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc47764cc9044ff095f74d97abaa437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.118748 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc47764cc9044ff095f74d97abaa437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.130293 29303 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Bootstrap starting.
I20260928 16:43:28.133895 29304 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Bootstrap starting.
I20260928 16:43:28.135667 29303 tablet_bootstrap.cc:654] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.135761 29305 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Bootstrap starting.
I20260928 16:43:28.136968 29303 log.cc:824] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:28.137728 29304 tablet_bootstrap.cc:654] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.139020 29304 log.cc:824] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:28.139804 29305 tablet_bootstrap.cc:654] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.139861 29303 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: No bootstrap required, opened a new log
I20260928 16:43:28.140146 29303 ts_tablet_manager.cc:1397] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:43:28.140976 29305 log.cc:824] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:28.142150 29304 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: No bootstrap required, opened a new log
I20260928 16:43:28.142427 29304 ts_tablet_manager.cc:1397] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:43:28.144562 29305 tablet_bootstrap.cc:492] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: No bootstrap required, opened a new log
I20260928 16:43:28.144855 29305 ts_tablet_manager.cc:1397] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 16:43:28.148584 29303 raft_consensus.cc:348] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.148998 29303 raft_consensus.cc:374] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.149101 29303 raft_consensus.cc:729] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851722a3ca344203a371b0da8b397480, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.149731 29303 consensus_queue.cc:260] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.150877 29284 heartbeater.cc:503] Master 127.26.101.62:45339 was elected leader, sending a full tablet report...
I20260928 16:43:28.151660 29303 ts_tablet_manager.cc:1428] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 16:43:28.153045 29304 raft_consensus.cc:348] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.153537 29304 raft_consensus.cc:374] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.153640 29304 raft_consensus.cc:729] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.154644 29304 consensus_queue.cc:260] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.155380 29305 raft_consensus.cc:348] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.157194 29305 raft_consensus.cc:374] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.157297 29305 raft_consensus.cc:729] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f774b42c344a48698388dca00aecc222, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.158043 29305 consensus_queue.cc:260] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.160175 29144 heartbeater.cc:503] Master 127.26.101.62:45339 was elected leader, sending a full tablet report...
I20260928 16:43:28.160529 29304 ts_tablet_manager.cc:1428] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 16:43:28.161348 29305 ts_tablet_manager.cc:1428] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
W20260928 16:43:28.213711 29004 tablet.cc:2369] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:28.237275 29145 tablet.cc:2369] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:28.261554 29285 tablet.cc:2369] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:28.413895 29311 raft_consensus.cc:482] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:28.414235 29311 raft_consensus.cc:504] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.416077 29311 leader_election.cc:288] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165), 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331)
I20260928 16:43:28.424432 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.416651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36083 (local address 127.26.101.1:34331)
0928 16:43:28.417101 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.417108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.417705 (+ 597us) server_negotiation.cc:407] Connection header received
0928 16:43:28.417895 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.417901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.417976 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.418047 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.418657 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.419641 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.422145 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.423370 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.423732 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.424003 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.424060 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.424191 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260928 16:43:28.424530 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.416597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:34331 (local address 127.26.101.2:36083)
0928 16:43:28.417591 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.417610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.417748 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.418119 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.418125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.418147 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.418498 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.418507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.419854 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.419859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.421950 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.421969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.423644 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.423652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.423839 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.423882 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.423990 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.424117 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":26,"thread_start_us":667,"threads_started":1}
I20260928 16:43:28.425107 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.416325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.26.101.2:37639)
0928 16:43:28.417248 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.417266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.417446 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.417808 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.417813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.417838 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.418140 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.418151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.419490 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.419495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.421950 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.421969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.423946 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.423953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.424148 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.424182 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.424280 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.424427 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":55,"thread_start_us":598,"threads_started":1}
I20260928 16:43:28.425173 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.416603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:37639 (local address 127.26.101.3:43165)
0928 16:43:28.416902 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.416908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.417380 (+ 472us) server_negotiation.cc:407] Connection header received
0928 16:43:28.417591 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.417595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.417657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.417748 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.418309 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.419333 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.422144 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.423390 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.423627 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.423712 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.424348 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.424918 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 16:43:28.425851 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc47764cc9044ff095f74d97abaa437d" candidate_uuid: "f774b42c344a48698388dca00aecc222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8" is_pre_election: true
I20260928 16:43:28.426030 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc47764cc9044ff095f74d97abaa437d" candidate_uuid: "f774b42c344a48698388dca00aecc222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480" is_pre_election: true
I20260928 16:43:28.426376 28953 raft_consensus.cc:2454] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f774b42c344a48698388dca00aecc222 in term 0.
I20260928 16:43:28.426478 29233 raft_consensus.cc:2454] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f774b42c344a48698388dca00aecc222 in term 0.
I20260928 16:43:28.427042 29024 leader_election.cc:302] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:28.427532 29311 raft_consensus.cc:2781] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:28.427673 29311 raft_consensus.cc:482] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:28.427778 29311 raft_consensus.cc:3037] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.430032 29311 raft_consensus.cc:504] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.430642 29311 leader_election.cc:288] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [CANDIDATE]: Term 1 election: Requested vote from peers 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165), 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331)
I20260928 16:43:28.431232 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc47764cc9044ff095f74d97abaa437d" candidate_uuid: "f774b42c344a48698388dca00aecc222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480"
I20260928 16:43:28.431311 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc47764cc9044ff095f74d97abaa437d" candidate_uuid: "f774b42c344a48698388dca00aecc222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8"
I20260928 16:43:28.431483 29233 raft_consensus.cc:3037] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.431536 28953 raft_consensus.cc:3037] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.433693 28953 raft_consensus.cc:2454] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f774b42c344a48698388dca00aecc222 in term 1.
I20260928 16:43:28.433667 29233 raft_consensus.cc:2454] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f774b42c344a48698388dca00aecc222 in term 1.
I20260928 16:43:28.434114 29027 leader_election.cc:302] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851722a3ca344203a371b0da8b397480, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:28.434463 29311 raft_consensus.cc:2781] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:28.435043 29311 raft_consensus.cc:686] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 1 LEADER]: Becoming Leader. State: Replica: f774b42c344a48698388dca00aecc222, State: Running, Role: LEADER
I20260928 16:43:28.435617 29311 consensus_queue.cc:237] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } }
I20260928 16:43:28.440188 28795 catalog_manager.cc:5495] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 reported cstate change: term changed from 0 to 1, leader changed from <none> to f774b42c344a48698388dca00aecc222 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "f774b42c344a48698388dca00aecc222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } health_report { overall_health: HEALTHY } } }
Sep 28 16:43:28 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613806, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:28.506983 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.493584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.26.101.62:45339)
0928 16:43:28.493781 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.493793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.494420 (+ 627us) server_negotiation.cc:407] Connection header received
0928 16:43:28.494590 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.494594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.494663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.494756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:28.496057 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.496786 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.498014 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.498448 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.498555 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.501924 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:28.501929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:28.501932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:28.501962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:28.504281 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:28.504966 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:28.504972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:28.504975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:28.505094 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:28.505455 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:28.505460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:28.505463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:28.505795 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:28.505926 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.506360 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.506673 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":80}
I20260928 16:43:28.507014 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.493687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45339 (local address 127.0.0.1:33550)
0928 16:43:28.494303 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.494319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.494475 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.494807 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.494811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.495530 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:28.495891 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.495903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.496917 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.496921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.497875 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.497887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.498630 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.498636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.498788 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.499447 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:28.499469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:28.501747 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:28.504415 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:28.504430 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:28.504439 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:28.504850 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:28.505200 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:28.505206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:28.505210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:28.505352 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:28.505926 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:28.505935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:28.506100 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.506242 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.506528 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":304,"threads_started":1}
I20260928 16:43:28.512435 28795 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:33550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:43:28.513525 28795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:28.520434 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 23b49b0b2cea419ca8a59876aaf7be0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adf5df5ef1040f89f3c58dd9236eb8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:28.520227 28933 tablet_service.cc:1467] Processing CreateTablet for tablet 23b49b0b2cea419ca8a59876aaf7be0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adf5df5ef1040f89f3c58dd9236eb8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:28.520304 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 23b49b0b2cea419ca8a59876aaf7be0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adf5df5ef1040f89f3c58dd9236eb8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:43:28.521250 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b49b0b2cea419ca8a59876aaf7be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.521299 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b49b0b2cea419ca8a59876aaf7be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.521291 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b49b0b2cea419ca8a59876aaf7be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.526563 29303 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: Bootstrap starting.
I20260928 16:43:28.528167 29303 tablet_bootstrap.cc:654] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.528813 29305 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: Bootstrap starting.
I20260928 16:43:28.530001 29304 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: Bootstrap starting.
I20260928 16:43:28.530591 29305 tablet_bootstrap.cc:654] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.531705 29304 tablet_bootstrap.cc:654] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.531883 29303 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: No bootstrap required, opened a new log
I20260928 16:43:28.532047 29303 ts_tablet_manager.cc:1397] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 16:43:28.532826 29303 raft_consensus.cc:348] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.533118 29303 raft_consensus.cc:374] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.533234 29303 raft_consensus.cc:729] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851722a3ca344203a371b0da8b397480, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.533475 29303 consensus_queue.cc:260] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.534078 29303 ts_tablet_manager.cc:1428] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 16:43:28.533967 29309 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.534279 29309 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.540617 29305 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: No bootstrap required, opened a new log
I20260928 16:43:28.540830 29305 ts_tablet_manager.cc:1397] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260928 16:43:28.540997 29304 tablet_bootstrap.cc:492] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: No bootstrap required, opened a new log
I20260928 16:43:28.541167 29304 ts_tablet_manager.cc:1397] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260928 16:43:28.541638 29305 raft_consensus.cc:348] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.541889 29305 raft_consensus.cc:374] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.541987 29305 raft_consensus.cc:729] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f774b42c344a48698388dca00aecc222, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.541939 29304 raft_consensus.cc:348] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.542163 29304 raft_consensus.cc:374] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.542264 29304 raft_consensus.cc:729] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.542214 29305 consensus_queue.cc:260] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.542487 29304 consensus_queue.cc:260] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.542861 29305 ts_tablet_manager.cc:1428] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 16:43:28.542698 29311 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.543023 29311 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.543036 29304 ts_tablet_manager.cc:1428] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 16:43:28.543244 29310 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.543601 29310 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.699296 29309 raft_consensus.cc:482] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:28.699519 29309 raft_consensus.cc:504] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.701119 29309 leader_election.cc:288] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:28.707928 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.701477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:49071 (local address 127.26.101.1:34331)
0928 16:43:28.701664 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.701670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.702303 (+ 633us) server_negotiation.cc:407] Connection header received
0928 16:43:28.702620 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.702626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.702706 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.702800 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.703532 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.704310 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.706109 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.706927 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.707062 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.707159 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.707522 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.707660 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:43:28.708143 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.701577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36633 (local address 127.26.101.2:46165)
0928 16:43:28.702729 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.702735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.702752 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:43:28.702818 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.702822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.702903 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.702997 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:28.703720 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.704505 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.706272 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.707177 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.707337 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.707415 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.707695 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.707832 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":932,"threads_started":1}
I20260928 16:43:28.707962 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.701369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:34331 (local address 127.26.101.3:49071)
0928 16:43:28.702177 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.702201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.702436 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.703024 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.703028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.703052 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.703357 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.703369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.704485 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.704490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.705933 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.705946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.707145 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.707151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.707345 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.707384 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.707479 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.707585 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":520,"threads_started":1}
I20260928 16:43:28.708180 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.701471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:46165 (local address 127.26.101.3:36633)
0928 16:43:28.702487 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.702511 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.702629 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.703024 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.703029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.703119 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:28.703494 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.703507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.704656 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.704659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.706096 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.706110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.707368 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.707373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.707518 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.707560 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.707646 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.707781 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":1,"spinlock_wait_cycles":29056,"thread_start_us":730,"threads_started":1}
I20260928 16:43:28.708959 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b49b0b2cea419ca8a59876aaf7be0f" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8" is_pre_election: true
I20260928 16:43:28.709240 28953 raft_consensus.cc:2454] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 0.
I20260928 16:43:28.709164 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b49b0b2cea419ca8a59876aaf7be0f" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222" is_pre_election: true
I20260928 16:43:28.709553 29093 raft_consensus.cc:2454] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 0.
I20260928 16:43:28.709715 29164 leader_election.cc:302] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, 851722a3ca344203a371b0da8b397480; no voters:
I20260928 16:43:28.710116 29309 raft_consensus.cc:2781] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:28.710232 29309 raft_consensus.cc:482] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:28.710330 29309 raft_consensus.cc:3037] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.712563 29309 raft_consensus.cc:504] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.713147 29309 leader_election.cc:288] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Requested vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:28.713677 28953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b49b0b2cea419ca8a59876aaf7be0f" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8"
I20260928 16:43:28.713912 28953 raft_consensus.cc:3037] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.715291 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b49b0b2cea419ca8a59876aaf7be0f" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222"
I20260928 16:43:28.715481 29093 raft_consensus.cc:3037] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:28.717537 29093 raft_consensus.cc:2454] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 1.
I20260928 16:43:28.717978 29165 leader_election.cc:302] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851722a3ca344203a371b0da8b397480, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:28.718272 29309 raft_consensus.cc:2781] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:28.718693 29309 raft_consensus.cc:686] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [term 1 LEADER]: Becoming Leader. State: Replica: 851722a3ca344203a371b0da8b397480, State: Running, Role: LEADER
I20260928 16:43:28.719099 28953 raft_consensus.cc:2454] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 1.
I20260928 16:43:28.719300 29309 consensus_queue.cc:237] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.723016 29328 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: TxnStatusTablet state changed. Reason: New leader 851722a3ca344203a371b0da8b397480. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.723412 29328 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:28.724107 29330 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:43:28.724388 29330 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:43:28.724257 28794 catalog_manager.cc:5495] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 reported cstate change: term changed from 0 to 1, leader changed from <none> to 851722a3ca344203a371b0da8b397480 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } health_report { overall_health: HEALTHY } } }
I20260928 16:43:28.784775 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.778007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.0.0.1:50526)
0928 16:43:28.778273 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.778294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.778465 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.778961 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.778966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.778978 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:28.779189 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.779200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.780778 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.780782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.781920 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.781935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.782940 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.782945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.783129 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.783199 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.784262 (+ 1063us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.784286 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.784371 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.784498 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 16:43:28.784816 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.778147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.26.101.3:43165)
0928 16:43:28.778348 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.778354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.778373 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:43:28.778680 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.778684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.778785 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.778919 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:28.779359 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.780606 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.782067 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.782763 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.782896 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.783393 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.784123 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.784285 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.784406 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.784498 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:43:28.793078 29093 raft_consensus.cc:1261] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Refusing update from remote peer 851722a3ca344203a371b0da8b397480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:28.793155 28953 raft_consensus.cc:1261] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Refusing update from remote peer 851722a3ca344203a371b0da8b397480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:28.794159 29328 consensus_queue.cc:1035] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:28.794507 29309 consensus_queue.cc:1035] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:28.802819 29311 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: TxnStatusTablet state changed. Reason: New leader 851722a3ca344203a371b0da8b397480. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.803280 29311 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.807207 29334 mvcc.cc:205] Tried to move back new op lower bound from 7334354160805298176 to 7334354160518111232. Current Snapshot: MvccSnapshot[applied={T|T < 7334354160805298176}]
I20260928 16:43:28.808965 29311 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.809299 29311 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P f774b42c344a48698388dca00aecc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.811204 29333 mvcc.cc:205] Tried to move back new op lower bound from 7334354160805298176 to 7334354160518111232. Current Snapshot: MvccSnapshot[applied={T|T < 7334354160805298176}]
I20260928 16:43:28.813421 29310 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: TxnStatusTablet state changed. Reason: New leader 851722a3ca344203a371b0da8b397480. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.813678 29310 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.818480 29309 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.818799 29309 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:28.820468 29310 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.820772 29310 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 5add1b4b69004a26b71d68b631804fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:43:28.823665 29328 tablet_replica.cc:439] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } } }
I20260928 16:43:28.823946 29328 tablet_replica.cc:442] T 23b49b0b2cea419ca8a59876aaf7be0f P 851722a3ca344203a371b0da8b397480: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:43:28.833894 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.826773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.26.101.2:46165)
0928 16:43:28.827023 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.827031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.827645 (+ 614us) server_negotiation.cc:407] Connection header received
0928 16:43:28.827880 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.827885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.828006 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.828167 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:28.828532 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.829719 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.831151 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.831793 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.831928 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.832372 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.833093 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.833240 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.833372 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.833462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:28.851473 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.845779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:55691 (local address 127.26.101.3:43165)
0928 16:43:28.845994 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.846005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.846253 (+ 248us) server_negotiation.cc:407] Connection header received
0928 16:43:28.846460 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.846464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.846540 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.846625 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:28.847130 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.848044 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.849542 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.850047 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.850175 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.850702 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.850812 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.850888 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.851050 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.851162 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:43:28.851560 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.845592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.26.101.2:55691)
0928 16:43:28.846129 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.846145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.846319 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.846656 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.846661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.846674 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:28.846957 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.846966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.848189 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.848194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.849388 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.849403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.850273 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.850278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.850404 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.850466 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.850916 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.850928 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.850997 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.851190 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":417,"threads_started":1}
I20260928 16:43:28.862066 28953 raft_consensus.cc:1261] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Refusing update from remote peer f774b42c344a48698388dca00aecc222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:28.862452 29233 raft_consensus.cc:1261] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Refusing update from remote peer f774b42c344a48698388dca00aecc222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:28.862969 29314 consensus_queue.cc:1035] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:28.863453 29311 consensus_queue.cc:1035] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:28.901397 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.894469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:46165 (local address 127.26.101.3:38823)
0928 16:43:28.895271 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:28.895297 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:28.895517 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:28.895873 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:28.895877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:28.895891 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:28.896332 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.896345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.897489 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.897495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.898702 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:28.898726 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.899726 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.899733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:28.899920 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.900015 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.900538 (+ 523us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.900551 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 16:43:28.900633 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:43:28.900956 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":714,"threads_started":1}
I20260928 16:43:28.901378 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:28.894452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:38823 (local address 127.26.101.2:46165)
0928 16:43:28.894659 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:28.894666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:28.895392 (+ 726us) server_negotiation.cc:407] Connection header received
0928 16:43:28.895672 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:28.895677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:28.895756 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:28.895831 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:28.896504 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.897356 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.898884 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:28.899527 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:28.899665 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:28.900283 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:28.900428 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:28.900513 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:28.900678 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 16:43:28.901096 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:43:28.943504 28795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:28.943813 28795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:28.947947 28795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Sending DeleteTablet for 3 replicas of tablet dc47764cc9044ff095f74d97abaa437d
I20260928 16:43:28.949278 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet dc47764cc9044ff095f74d97abaa437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:28 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260928 16:43:28.949282 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet dc47764cc9044ff095f74d97abaa437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:28 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:55644
I20260928 16:43:28.950661 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet dc47764cc9044ff095f74d97abaa437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:28 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51706
I20260928 16:43:28.955605 29358 tablet_replica.cc:330] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: stopping tablet replica
I20260928 16:43:28.956012 29358 raft_consensus.cc:2229] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:28.956480 29358 raft_consensus.cc:2258] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:28.958597 29358 ts_tablet_manager.cc:1905] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:28.958420 28795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:28.959534 28795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:28.967713 28932 tablet_service.cc:1467] Processing CreateTablet for tablet 1dccab5b33e74435862809c646ae6792 (DEFAULT_TABLE table=test-table [id=47c567ce0ca54b2cb398f9f0df462721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.968075 29072 tablet_service.cc:1467] Processing CreateTablet for tablet 1dccab5b33e74435862809c646ae6792 (DEFAULT_TABLE table=test-table [id=47c567ce0ca54b2cb398f9f0df462721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.968299 28932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dccab5b33e74435862809c646ae6792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.968660 29072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dccab5b33e74435862809c646ae6792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.969522 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 1dccab5b33e74435862809c646ae6792 (DEFAULT_TABLE table=test-table [id=47c567ce0ca54b2cb398f9f0df462721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:28.970011 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dccab5b33e74435862809c646ae6792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:28.975183 29305 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Bootstrap starting.
I20260928 16:43:28.975256 29304 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Bootstrap starting.
I20260928 16:43:28.976922 29305 tablet_bootstrap.cc:654] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.976995 29304 tablet_bootstrap.cc:654] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.980000 29359 tablet_replica.cc:330] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: stopping tablet replica
I20260928 16:43:28.980517 29359 raft_consensus.cc:2229] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:28.980726 29303 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Bootstrap starting.
I20260928 16:43:28.981432 29359 raft_consensus.cc:2258] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:28.982538 29303 tablet_bootstrap.cc:654] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:28.990447 29305 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: No bootstrap required, opened a new log
I20260928 16:43:28.990646 29305 ts_tablet_manager.cc:1397] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260928 16:43:28.991540 29305 raft_consensus.cc:348] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.991827 29305 raft_consensus.cc:374] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.991895 29305 raft_consensus.cc:729] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f774b42c344a48698388dca00aecc222, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.992178 29305 consensus_queue.cc:260] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.993682 29359 ts_tablet_manager.cc:1905] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:28.995577 29305 ts_tablet_manager.cc:1428] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:28.997236 29304 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: No bootstrap required, opened a new log
I20260928 16:43:28.997450 29304 ts_tablet_manager.cc:1397] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20260928 16:43:28.998389 29304 raft_consensus.cc:348] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:28.998693 29304 raft_consensus.cc:374] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:28.998778 29304 raft_consensus.cc:729] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Initialized, Role: FOLLOWER
I20260928 16:43:28.999214 29304 consensus_queue.cc:260] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.000078 29303 tablet_bootstrap.cc:492] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: No bootstrap required, opened a new log
I20260928 16:43:29.000310 29303 ts_tablet_manager.cc:1397] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20260928 16:43:29.000820 29358 ts_tablet_manager.cc:1918] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:29.001010 29358 log.cc:1196] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/dc47764cc9044ff095f74d97abaa437d
I20260928 16:43:29.001595 29358 ts_tablet_manager.cc:1939] T dc47764cc9044ff095f74d97abaa437d P 851722a3ca344203a371b0da8b397480: Deleting consensus metadata
I20260928 16:43:29.001456 29303 raft_consensus.cc:348] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.001863 29303 raft_consensus.cc:374] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:29.001997 29303 raft_consensus.cc:729] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851722a3ca344203a371b0da8b397480, State: Initialized, Role: FOLLOWER
I20260928 16:43:29.002328 29303 consensus_queue.cc:260] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.002640 29304 ts_tablet_manager.cc:1428] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:29.003470 28782 catalog_manager.cc:4841] TS 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165): tablet dc47764cc9044ff095f74d97abaa437d (table test-table [id=01a84d73ef1c428ca7cd94373ce1a744]) successfully deleted
I20260928 16:43:29.005600 29357 tablet_replica.cc:330] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: stopping tablet replica
I20260928 16:43:29.005997 29357 raft_consensus.cc:2229] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:29.006341 29357 raft_consensus.cc:2258] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:29.006479 29303 ts_tablet_manager.cc:1428] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260928 16:43:29.016628 29357 ts_tablet_manager.cc:1905] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:29.018008 29359 ts_tablet_manager.cc:1918] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:29.018208 29359 log.cc:1196] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/dc47764cc9044ff095f74d97abaa437d
I20260928 16:43:29.018724 29359 ts_tablet_manager.cc:1939] T dc47764cc9044ff095f74d97abaa437d P f774b42c344a48698388dca00aecc222: Deleting consensus metadata
I20260928 16:43:29.020557 28780 catalog_manager.cc:4841] TS f774b42c344a48698388dca00aecc222 (127.26.101.2:46165): tablet dc47764cc9044ff095f74d97abaa437d (table test-table [id=01a84d73ef1c428ca7cd94373ce1a744]) successfully deleted
I20260928 16:43:29.023263 29357 ts_tablet_manager.cc:1918] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:29.023454 29357 log.cc:1196] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/dc47764cc9044ff095f74d97abaa437d
I20260928 16:43:29.023978 29357 ts_tablet_manager.cc:1939] T dc47764cc9044ff095f74d97abaa437d P 5add1b4b69004a26b71d68b631804fc8: Deleting consensus metadata
I20260928 16:43:29.025677 28779 catalog_manager.cc:4841] TS 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331): tablet dc47764cc9044ff095f74d97abaa437d (table test-table [id=01a84d73ef1c428ca7cd94373ce1a744]) successfully deleted
I20260928 16:43:29.390686 29310 raft_consensus.cc:482] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:29.390951 29310 raft_consensus.cc:504] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.392627 29310 leader_election.cc:288] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f774b42c344a48698388dca00aecc222 (127.26.101.2:46165), 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165)
I20260928 16:43:29.402180 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.393081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57777 (local address 127.26.101.2:46165)
0928 16:43:29.393309 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:29.393316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:29.393880 (+ 564us) server_negotiation.cc:407] Connection header received
0928 16:43:29.394199 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:29.394205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:29.394291 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:29.394360 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:29.395157 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.396341 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.399326 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.400749 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.400925 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.401035 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:29.401667 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 16:43:29.401856 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:43:29.403043 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.392954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:46165 (local address 127.26.101.1:57777)
0928 16:43:29.393745 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.393768 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:29.394014 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:29.394641 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:29.394646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:29.394677 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:29.394948 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.394960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.396585 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.396591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.399003 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.399027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.401211 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.401219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.401453 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.401492 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:43:29.402504 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0928 16:43:29.402682 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"threads_started":1}
I20260928 16:43:29.403846 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222" is_pre_election: true
I20260928 16:43:29.404167 29093 raft_consensus.cc:2454] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 0.
I20260928 16:43:29.404228 29328 raft_consensus.cc:482] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:29.404398 29328 raft_consensus.cc:504] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.404688 28883 leader_election.cc:302] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:29.405009 29328 leader_election.cc:288] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:29.405086 29310 raft_consensus.cc:2781] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:29.405189 29310 raft_consensus.cc:482] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:29.405265 29310 raft_consensus.cc:3037] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:29.406174 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222" is_pre_election: true
I20260928 16:43:29.406392 29093 raft_consensus.cc:2454] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 0.
I20260928 16:43:29.406751 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8" is_pre_election: true
I20260928 16:43:29.407411 29165 leader_election.cc:302] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851722a3ca344203a371b0da8b397480, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:29.407732 29328 raft_consensus.cc:2781] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:29.407828 29328 raft_consensus.cc:482] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:29.407945 29328 raft_consensus.cc:3037] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:29.408476 29310 raft_consensus.cc:504] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.408903 28952 raft_consensus.cc:2379] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 851722a3ca344203a371b0da8b397480 in current term 1: Already voted for candidate 5add1b4b69004a26b71d68b631804fc8 in this term.
I20260928 16:43:29.410115 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.392983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.26.101.1:55119)
0928 16:43:29.403279 (+ 10296us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.403298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:29.403432 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:29.403760 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:29.403765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:29.403791 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:29.403954 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.403965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.405567 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.405572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.407485 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.407501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.409254 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.409262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.409470 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.409503 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:43:29.409659 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 16:43:29.409810 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10205,"thread_start_us":714,"threads_started":1}
I20260928 16:43:29.410598 29310 leader_election.cc:288] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 election: Requested vote from peers f774b42c344a48698388dca00aecc222 (127.26.101.2:46165), 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165)
I20260928 16:43:29.410820 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.393507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55119 (local address 127.26.101.3:43165)
0928 16:43:29.402967 (+ 9460us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:29.402973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:29.403456 (+ 483us) server_negotiation.cc:407] Connection header received
0928 16:43:29.403546 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:29.403552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:29.403629 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:29.403734 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:29.404131 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.405386 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.407658 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.408871 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.409047 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.409130 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:29.410327 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0928 16:43:29.410475 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9337,"thread_start_us":9229,"threads_started":1}
I20260928 16:43:29.411597 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480" is_pre_election: true
I20260928 16:43:29.411612 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222"
I20260928 16:43:29.411806 29093 raft_consensus.cc:3037] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:29.411262 29328 raft_consensus.cc:504] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.412235 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480"
I20260928 16:43:29.412348 29233 raft_consensus.cc:2379] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5add1b4b69004a26b71d68b631804fc8 in current term 1: Already voted for candidate 851722a3ca344203a371b0da8b397480 in this term.
I20260928 16:43:29.412916 29328 leader_election.cc:288] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Requested vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:29.413734 29092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222"
I20260928 16:43:29.414358 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dccab5b33e74435862809c646ae6792" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8"
I20260928 16:43:29.414587 28952 raft_consensus.cc:2379] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 851722a3ca344203a371b0da8b397480 in current term 1: Already voted for candidate 5add1b4b69004a26b71d68b631804fc8 in this term.
I20260928 16:43:29.414953 29093 raft_consensus.cc:2454] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 1.
I20260928 16:43:29.414988 29164 leader_election.cc:302] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 851722a3ca344203a371b0da8b397480; no voters: 5add1b4b69004a26b71d68b631804fc8, f774b42c344a48698388dca00aecc222
I20260928 16:43:29.415302 29328 raft_consensus.cc:2726] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:43:29.415402 28883 leader_election.cc:302] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, f774b42c344a48698388dca00aecc222; no voters: 851722a3ca344203a371b0da8b397480
I20260928 16:43:29.415738 29310 raft_consensus.cc:2781] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:29.416162 29310 raft_consensus.cc:686] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Running, Role: LEADER
I20260928 16:43:29.416561 29310 consensus_queue.cc:237] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:29.420305 28795 catalog_manager.cc:5495] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "5add1b4b69004a26b71d68b631804fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:29.451668 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.445320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.26.101.1:34331)
0928 16:43:29.445835 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:29.445839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:29.445911 (+ 72us) server_negotiation.cc:407] Connection header received
0928 16:43:29.446090 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:29.446093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:29.446171 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:29.446317 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:29.446718 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.447577 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.449075 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.449535 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.449646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.450129 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:29.450759 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:29.450943 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:29.451216 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 16:43:29.451329 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260928 16:43:29.467868 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.462364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:35481 (local address 127.26.101.3:43165)
0928 16:43:29.462536 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:29.462541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:29.462934 (+ 393us) server_negotiation.cc:407] Connection header received
0928 16:43:29.463115 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:29.463120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:29.463193 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:29.463263 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:29.463809 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.464570 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.466054 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.466603 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.466720 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.467231 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:29.467350 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:29.467428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:29.467580 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 16:43:29.467656 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:43:29.467921 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.462235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.26.101.1:35481)
0928 16:43:29.462834 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.462851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:29.462976 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:29.463317 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:29.463320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:29.463330 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:29.463602 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.463610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.464764 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.464768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.465908 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.465922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.466776 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.466782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.466909 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.466986 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:29.467445 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:29.467454 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:43:29.467510 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 16:43:29.467614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":274,"threads_started":1}
I20260928 16:43:29.476030 29093 raft_consensus.cc:1261] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Refusing update from remote peer 5add1b4b69004a26b71d68b631804fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:29.476150 29233 raft_consensus.cc:1261] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Refusing update from remote peer 5add1b4b69004a26b71d68b631804fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:29.476701 29366 consensus_queue.cc:1035] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:29.477051 29310 consensus_queue.cc:1035] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:29.487950 29369 mvcc.cc:205] Tried to move back new op lower bound from 7334354163607658496 to 7334354163374194688. Current Snapshot: MvccSnapshot[applied={T|T < 7334354163607658496}]
I20260928 16:43:29.523007 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.506403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:38615 (local address 127.26.101.1:34331)
0928 16:43:29.506564 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:29.506569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:29.517053 (+ 10484us) server_negotiation.cc:407] Connection header received
0928 16:43:29.517166 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:29.517172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:29.517264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:29.517363 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:29.517876 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.518956 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.520943 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.521558 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.521659 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.522235 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:29.522376 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:29.522469 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:29.522609 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:43:29.522699 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:43:29.523044 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:29.505748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:34331 (local address 127.26.101.3:38615)
0928 16:43:29.516500 (+ 10752us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:29.516521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:29.516671 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:29.517449 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:29.517454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:29.517469 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:29.517682 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.517691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.519204 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.519210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.520760 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:29.520777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:29.521818 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:29.521823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:29.521994 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:29.522046 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:29.522469 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:29.522481 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:43:29.522561 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:43:29.522718 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10593,"thread_start_us":11010,"threads_started":1}
W20260928 16:43:29.525426 29369 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:43:31.136106 28794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:31.136389 28794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:31.140965 28794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Sending DeleteTablet for 3 replicas of tablet 1dccab5b33e74435862809c646ae6792
I20260928 16:43:31.141940 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dccab5b33e74435862809c646ae6792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51706
I20260928 16:43:31.141947 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dccab5b33e74435862809c646ae6792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260928 16:43:31.142241 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dccab5b33e74435862809c646ae6792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:55644
I20260928 16:43:31.154736 29412 tablet_replica.cc:330] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: stopping tablet replica
I20260928 16:43:31.155558 29410 tablet_replica.cc:330] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: stopping tablet replica
I20260928 16:43:31.155637 29411 tablet_replica.cc:330] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: stopping tablet replica
I20260928 16:43:31.155673 29412 raft_consensus.cc:2229] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:31.155776 29410 raft_consensus.cc:2229] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:31.155850 29411 raft_consensus.cc:2229] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:31.156024 29410 raft_consensus.cc:2258] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.156091 29411 raft_consensus.cc:2258] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.156280 29412 raft_consensus.cc:2258] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.157083 29410 ts_tablet_manager.cc:1905] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.157367 29411 ts_tablet_manager.cc:1905] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.157424 29412 ts_tablet_manager.cc:1905] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.163928 29410 ts_tablet_manager.cc:1918] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:31.164036 29412 ts_tablet_manager.cc:1918] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:31.164129 29411 ts_tablet_manager.cc:1918] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:31.164152 29410 log.cc:1196] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/1dccab5b33e74435862809c646ae6792
I20260928 16:43:31.164183 29412 log.cc:1196] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/1dccab5b33e74435862809c646ae6792
I20260928 16:43:31.164259 29411 log.cc:1196] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/1dccab5b33e74435862809c646ae6792
I20260928 16:43:31.164654 29412 ts_tablet_manager.cc:1939] T 1dccab5b33e74435862809c646ae6792 P 5add1b4b69004a26b71d68b631804fc8: Deleting consensus metadata
I20260928 16:43:31.164681 29411 ts_tablet_manager.cc:1939] T 1dccab5b33e74435862809c646ae6792 P f774b42c344a48698388dca00aecc222: Deleting consensus metadata
I20260928 16:43:31.164677 29410 ts_tablet_manager.cc:1939] T 1dccab5b33e74435862809c646ae6792 P 851722a3ca344203a371b0da8b397480: Deleting consensus metadata
I20260928 16:43:31.165867 28780 catalog_manager.cc:4841] TS f774b42c344a48698388dca00aecc222 (127.26.101.2:46165): tablet 1dccab5b33e74435862809c646ae6792 (table test-table [id=47c567ce0ca54b2cb398f9f0df462721]) successfully deleted
I20260928 16:43:31.165913 28782 catalog_manager.cc:4841] TS 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165): tablet 1dccab5b33e74435862809c646ae6792 (table test-table [id=47c567ce0ca54b2cb398f9f0df462721]) successfully deleted
I20260928 16:43:31.165874 28779 catalog_manager.cc:4841] TS 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331): tablet 1dccab5b33e74435862809c646ae6792 (table test-table [id=47c567ce0ca54b2cb398f9f0df462721]) successfully deleted
Sep 28 16:43:31 dist-test-slave-t0tw krb5kdc[28749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:43:31 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:31.200941 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.186017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.26.101.62:45339)
0928 16:43:31.186546 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:31.186550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:31.186574 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:43:31.186710 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:31.186713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:31.186778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:31.186895 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:31.188416 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.189245 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.190796 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.191290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.191409 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:31.195557 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:31.195562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:31.195565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:31.195605 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:31.198273 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:31.199122 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:31.199128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:31.199130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:31.199262 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:31.199612 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:31.199615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:31.199617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:31.199927 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:31.200117 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:31.200458 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 16:43:31.200657 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":276,"threads_started":1}
I20260928 16:43:31.204491 28794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:31.205103 28794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:31.211628 28933 tablet_service.cc:1467] Processing CreateTablet for tablet 28b8972ba95942408615a2d7518fe962 (DEFAULT_TABLE table=test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.211792 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 28b8972ba95942408615a2d7518fe962 (DEFAULT_TABLE table=test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.211870 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 28b8972ba95942408615a2d7518fe962 (DEFAULT_TABLE table=test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.212123 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b8972ba95942408615a2d7518fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.212188 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b8972ba95942408615a2d7518fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.212340 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b8972ba95942408615a2d7518fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.218045 29421 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Bootstrap starting.
I20260928 16:43:31.219126 29423 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Bootstrap starting.
I20260928 16:43:31.220448 29421 tablet_bootstrap.cc:654] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.220960 29423 tablet_bootstrap.cc:654] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.226781 29422 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Bootstrap starting.
I20260928 16:43:31.228205 29421 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: No bootstrap required, opened a new log
I20260928 16:43:31.228436 29421 ts_tablet_manager.cc:1397] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260928 16:43:31.228492 29423 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: No bootstrap required, opened a new log
I20260928 16:43:31.228714 29423 ts_tablet_manager.cc:1397] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 16:43:31.229415 29422 tablet_bootstrap.cc:654] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.229435 29421 raft_consensus.cc:348] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.229744 29421 raft_consensus.cc:374] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.229676 29423 raft_consensus.cc:348] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.229856 29421 raft_consensus.cc:729] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851722a3ca344203a371b0da8b397480, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.229930 29423 raft_consensus.cc:374] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.230026 29423 raft_consensus.cc:729] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f774b42c344a48698388dca00aecc222, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.230099 29421 consensus_queue.cc:260] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.230268 29423 consensus_queue.cc:260] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.231148 29421 ts_tablet_manager.cc:1428] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:31.232929 29423 ts_tablet_manager.cc:1428] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 16:43:31.233618 29422 tablet_bootstrap.cc:492] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: No bootstrap required, opened a new log
I20260928 16:43:31.233788 29422 ts_tablet_manager.cc:1397] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:43:31.234673 29422 raft_consensus.cc:348] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.234917 29422 raft_consensus.cc:374] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.235020 29422 raft_consensus.cc:729] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.235262 29422 consensus_queue.cc:260] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.235702 29422 ts_tablet_manager.cc:1428] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:43:31.398136 29383 raft_consensus.cc:482] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:31.398438 29383 raft_consensus.cc:504] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.399392 29383 leader_election.cc:288] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f774b42c344a48698388dca00aecc222 (127.26.101.2:46165), 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165)
I20260928 16:43:31.400053 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b8972ba95942408615a2d7518fe962" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222" is_pre_election: true
I20260928 16:43:31.400063 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b8972ba95942408615a2d7518fe962" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480" is_pre_election: true
I20260928 16:43:31.400315 29093 raft_consensus.cc:2454] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 0.
I20260928 16:43:31.400334 29233 raft_consensus.cc:2454] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 0.
I20260928 16:43:31.400935 28883 leader_election.cc:302] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:31.401268 29383 raft_consensus.cc:2781] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:31.401386 29383 raft_consensus.cc:482] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:31.401479 29383 raft_consensus.cc:3037] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.403759 29383 raft_consensus.cc:504] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.404373 29383 leader_election.cc:288] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 election: Requested vote from peers f774b42c344a48698388dca00aecc222 (127.26.101.2:46165), 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165)
I20260928 16:43:31.404899 29093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b8972ba95942408615a2d7518fe962" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222"
I20260928 16:43:31.404923 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b8972ba95942408615a2d7518fe962" candidate_uuid: "5add1b4b69004a26b71d68b631804fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851722a3ca344203a371b0da8b397480"
I20260928 16:43:31.405133 29093 raft_consensus.cc:3037] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.405187 29233 raft_consensus.cc:3037] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.407282 29233 raft_consensus.cc:2454] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 1.
I20260928 16:43:31.407307 29093 raft_consensus.cc:2454] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5add1b4b69004a26b71d68b631804fc8 in term 1.
I20260928 16:43:31.407768 28885 leader_election.cc:302] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, 851722a3ca344203a371b0da8b397480; no voters:
I20260928 16:43:31.408082 29383 raft_consensus.cc:2781] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:31.408224 29383 raft_consensus.cc:686] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Running, Role: LEADER
I20260928 16:43:31.408483 29383 consensus_queue.cc:237] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.411325 28794 catalog_manager.cc:5495] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "5add1b4b69004a26b71d68b631804fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:31.469810 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.464590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.26.101.1:34331)
0928 16:43:31.465148 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:31.465152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:31.465173 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:43:31.465244 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:31.465248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:31.465310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:31.465391 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:31.465807 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.466683 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.468132 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.468568 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.468663 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:31.469038 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:31.469178 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:31.469262 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:31.469381 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:43:31.469447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":305,"threads_started":1}
I20260928 16:43:31.478680 29233 raft_consensus.cc:1261] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Refusing update from remote peer 5add1b4b69004a26b71d68b631804fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:31.478704 29093 raft_consensus.cc:1261] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Refusing update from remote peer 5add1b4b69004a26b71d68b631804fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:31.479338 29383 consensus_queue.cc:1035] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:31.479637 29404 consensus_queue.cc:1035] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:31.531658 28794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:31.532032 28794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:31.535223 28794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Sending DeleteTablet for 3 replicas of tablet 28b8972ba95942408615a2d7518fe962
I20260928 16:43:31.536108 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b8972ba95942408615a2d7518fe962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:55644
I20260928 16:43:31.536418 29412 tablet_replica.cc:330] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: stopping tablet replica
I20260928 16:43:31.536374 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b8972ba95942408615a2d7518fe962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51706
I20260928 16:43:31.536471 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b8972ba95942408615a2d7518fe962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:31 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260928 16:43:31.536700 29411 tablet_replica.cc:330] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: stopping tablet replica
I20260928 16:43:31.536670 29412 raft_consensus.cc:2229] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:31.536739 29410 tablet_replica.cc:330] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: stopping tablet replica
I20260928 16:43:31.536914 29410 raft_consensus.cc:2229] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:31.536947 29411 raft_consensus.cc:2229] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:31.537065 29412 raft_consensus.cc:2258] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.537091 29410 raft_consensus.cc:2258] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.537170 29411 raft_consensus.cc:2258] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:31.537405 28794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:31.538090 28794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:31.538139 29412 ts_tablet_manager.cc:1905] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.538378 29410 ts_tablet_manager.cc:1905] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.540138 29411 ts_tablet_manager.cc:1905] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:31.544962 29410 ts_tablet_manager.cc:1918] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:31.544962 29412 ts_tablet_manager.cc:1918] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:31.545130 29412 log.cc:1196] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/28b8972ba95942408615a2d7518fe962
I20260928 16:43:31.545132 29410 log.cc:1196] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/28b8972ba95942408615a2d7518fe962
I20260928 16:43:31.545537 29412 ts_tablet_manager.cc:1939] T 28b8972ba95942408615a2d7518fe962 P 5add1b4b69004a26b71d68b631804fc8: Deleting consensus metadata
I20260928 16:43:31.545596 29410 ts_tablet_manager.cc:1939] T 28b8972ba95942408615a2d7518fe962 P 851722a3ca344203a371b0da8b397480: Deleting consensus metadata
I20260928 16:43:31.546577 28782 catalog_manager.cc:4841] TS 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165): tablet 28b8972ba95942408615a2d7518fe962 (table test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]) successfully deleted
I20260928 16:43:31.546753 28779 catalog_manager.cc:4841] TS 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331): tablet 28b8972ba95942408615a2d7518fe962 (table test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]) successfully deleted
I20260928 16:43:31.547489 29411 ts_tablet_manager.cc:1918] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:43:31.547677 29411 log.cc:1196] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/28b8972ba95942408615a2d7518fe962
I20260928 16:43:31.548117 29411 ts_tablet_manager.cc:1939] T 28b8972ba95942408615a2d7518fe962 P f774b42c344a48698388dca00aecc222: Deleting consensus metadata
I20260928 16:43:31.548702 28933 tablet_service.cc:1467] Processing CreateTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 (DEFAULT_TABLE table=test-table [id=39431c34bef44daeb84ddde0b0e62a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.548885 29073 tablet_service.cc:1467] Processing CreateTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 (DEFAULT_TABLE table=test-table [id=39431c34bef44daeb84ddde0b0e62a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.548889 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 (DEFAULT_TABLE table=test-table [id=39431c34bef44daeb84ddde0b0e62a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:31.549124 28933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b2f09c8d604ddd8e189a525eb251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.549208 28780 catalog_manager.cc:4841] TS f774b42c344a48698388dca00aecc222 (127.26.101.2:46165): tablet 28b8972ba95942408615a2d7518fe962 (table test-table [id=ab2d5339cd0a4bd4aa76aa2290a2b5ed]) successfully deleted
I20260928 16:43:31.549281 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b2f09c8d604ddd8e189a525eb251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.549281 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b2f09c8d604ddd8e189a525eb251f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:31.554523 29422 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Bootstrap starting.
I20260928 16:43:31.555557 29423 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Bootstrap starting.
I20260928 16:43:31.556238 29422 tablet_bootstrap.cc:654] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.556393 29421 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Bootstrap starting.
I20260928 16:43:31.557183 29423 tablet_bootstrap.cc:654] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.558048 29421 tablet_bootstrap.cc:654] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:31.559805 29422 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: No bootstrap required, opened a new log
I20260928 16:43:31.559813 29423 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: No bootstrap required, opened a new log
I20260928 16:43:31.559962 29422 ts_tablet_manager.cc:1397] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 16:43:31.559985 29423 ts_tablet_manager.cc:1397] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 16:43:31.560627 29423 raft_consensus.cc:348] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.560783 29422 raft_consensus.cc:348] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.560935 29423 raft_consensus.cc:374] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.560961 29421 tablet_bootstrap.cc:492] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: No bootstrap required, opened a new log
I20260928 16:43:31.561033 29422 raft_consensus.cc:374] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.561102 29423 raft_consensus.cc:729] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f774b42c344a48698388dca00aecc222, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.561127 29421 ts_tablet_manager.cc:1397] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:43:31.561131 29422 raft_consensus.cc:729] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5add1b4b69004a26b71d68b631804fc8, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.561399 29422 consensus_queue.cc:260] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.561474 29423 consensus_queue.cc:260] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.562032 29422 ts_tablet_manager.cc:1428] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 16:43:31.562206 29423 ts_tablet_manager.cc:1428] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:43:31.561977 29421 raft_consensus.cc:348] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.562392 29421 raft_consensus.cc:374] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:31.562503 29421 raft_consensus.cc:729] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851722a3ca344203a371b0da8b397480, State: Initialized, Role: FOLLOWER
I20260928 16:43:31.562788 29421 consensus_queue.cc:260] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.563441 29421 ts_tablet_manager.cc:1428] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 16:43:31.836926 29391 raft_consensus.cc:482] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:31.837146 29391 raft_consensus.cc:504] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.837819 29391 leader_election.cc:288] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:31.838449 29092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b2f09c8d604ddd8e189a525eb251f1" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222" is_pre_election: true
I20260928 16:43:31.838514 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b2f09c8d604ddd8e189a525eb251f1" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8" is_pre_election: true
I20260928 16:43:31.838716 29092 raft_consensus.cc:2454] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 0.
I20260928 16:43:31.838766 28952 raft_consensus.cc:2454] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 0.
I20260928 16:43:31.839234 29165 leader_election.cc:302] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851722a3ca344203a371b0da8b397480, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:31.839507 29391 raft_consensus.cc:2781] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:31.839613 29391 raft_consensus.cc:482] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:31.839700 29391 raft_consensus.cc:3037] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.842063 29391 raft_consensus.cc:504] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.842677 29391 leader_election.cc:288] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Requested vote from peers 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331), f774b42c344a48698388dca00aecc222 (127.26.101.2:46165)
I20260928 16:43:31.843262 29092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b2f09c8d604ddd8e189a525eb251f1" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f774b42c344a48698388dca00aecc222"
I20260928 16:43:31.843344 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b2f09c8d604ddd8e189a525eb251f1" candidate_uuid: "851722a3ca344203a371b0da8b397480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5add1b4b69004a26b71d68b631804fc8"
I20260928 16:43:31.843484 29092 raft_consensus.cc:3037] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.843545 28952 raft_consensus.cc:3037] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:31.845506 28952 raft_consensus.cc:2454] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 1.
I20260928 16:43:31.845501 29092 raft_consensus.cc:2454] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851722a3ca344203a371b0da8b397480 in term 1.
I20260928 16:43:31.845940 29164 leader_election.cc:302] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5add1b4b69004a26b71d68b631804fc8, 851722a3ca344203a371b0da8b397480, f774b42c344a48698388dca00aecc222; no voters:
I20260928 16:43:31.846303 29391 raft_consensus.cc:2781] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:31.846446 29391 raft_consensus.cc:686] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 1 LEADER]: Becoming Leader. State: Replica: 851722a3ca344203a371b0da8b397480, State: Running, Role: LEADER
I20260928 16:43:31.846698 29391 consensus_queue.cc:237] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } }
I20260928 16:43:31.849452 28794 catalog_manager.cc:5495] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 reported cstate change: term changed from 0 to 1, leader changed from <none> to 851722a3ca344203a371b0da8b397480 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "851722a3ca344203a371b0da8b397480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851722a3ca344203a371b0da8b397480" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43165 } health_report { overall_health: HEALTHY } } }
I20260928 16:43:31.939287 29439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.933438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.26.101.3:43165)
0928 16:43:31.934078 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:31.934083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:31.934103 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:43:31.934163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:31.934166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:31.934227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:31.934312 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:31.934918 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.935929 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.937386 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.937831 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.937947 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:31.938432 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:31.938560 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:31.938641 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:31.938818 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 16:43:31.938903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":319,"threads_started":1}
I20260928 16:43:31.954912 29439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.949342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:51217 (local address 127.26.101.3:43165)
0928 16:43:31.949538 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:31.949545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:31.950042 (+ 497us) server_negotiation.cc:407] Connection header received
0928 16:43:31.950210 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:31.950214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:31.950270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:31.950340 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:31.950778 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.951592 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.952924 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.953396 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.953581 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:31.953975 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:31.954082 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:31.954159 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:31.954312 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:43:31.954447 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":59}
I20260928 16:43:31.954918 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:31.949102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43165 (local address 127.26.101.3:51217)
0928 16:43:31.949938 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:31.949954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:31.950096 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:31.950374 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:31.950378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:31.950388 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:43:31.950641 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:31.950650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.951712 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.951718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:31.952785 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:31.952798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:31.953583 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:31.953589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:31.953736 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:31.953792 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:43:31.954181 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:43:31.954193 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 16:43:31.954288 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:43:31.954512 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":41,"thread_start_us":458,"threads_started":1}
I20260928 16:43:31.964334 29092 raft_consensus.cc:1261] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Refusing update from remote peer 851722a3ca344203a371b0da8b397480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:31.964450 28952 raft_consensus.cc:1261] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Refusing update from remote peer 851722a3ca344203a371b0da8b397480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:31.965023 29405 consensus_queue.cc:1035] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5add1b4b69004a26b71d68b631804fc8" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:31.965474 29438 consensus_queue.cc:1035] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f774b42c344a48698388dca00aecc222" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:43:31.989676 29388 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:43:33.601943 28794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:33.602226 28794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:33.607431 28794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54ed1ff701af429eaa55159fab546208: Sending DeleteTablet for 3 replicas of tablet 04b2f09c8d604ddd8e189a525eb251f1
I20260928 16:43:33.608544 29073 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:51706
I20260928 16:43:33.608546 28933 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:55644
I20260928 16:43:33.608639 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b2f09c8d604ddd8e189a525eb251f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:33 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260928 16:43:33.609752 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28866
I20260928 16:43:33.619863 29466 tablet_replica.cc:330] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: stopping tablet replica
I20260928 16:43:33.620157 29466 raft_consensus.cc:2229] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:33.620451 29466 raft_consensus.cc:2258] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:33.621392 29465 tablet_replica.cc:330] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: stopping tablet replica
I20260928 16:43:33.621609 29467 tablet_replica.cc:330] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: stopping tablet replica
I20260928 16:43:33.621639 29465 raft_consensus.cc:2229] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:33.621833 29467 raft_consensus.cc:2229] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:33.621901 29465 raft_consensus.cc:2258] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:33.622392 29467 raft_consensus.cc:2258] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:33.622855 29466 ts_tablet_manager.cc:1905] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:33.623338 29467 ts_tablet_manager.cc:1905] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:33.626144 29465 ts_tablet_manager.cc:1905] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:33.630280 29466 ts_tablet_manager.cc:1918] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:33.630455 29466 log.cc:1196] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/04b2f09c8d604ddd8e189a525eb251f1
I20260928 16:43:33.630472 29467 ts_tablet_manager.cc:1918] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:33.630667 29467 log.cc:1196] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/04b2f09c8d604ddd8e189a525eb251f1
I20260928 16:43:33.630927 29466 ts_tablet_manager.cc:1939] T 04b2f09c8d604ddd8e189a525eb251f1 P f774b42c344a48698388dca00aecc222: Deleting consensus metadata
I20260928 16:43:33.631321 29467 ts_tablet_manager.cc:1939] T 04b2f09c8d604ddd8e189a525eb251f1 P 851722a3ca344203a371b0da8b397480: Deleting consensus metadata
I20260928 16:43:33.632246 28780 catalog_manager.cc:4841] TS f774b42c344a48698388dca00aecc222 (127.26.101.2:46165): tablet 04b2f09c8d604ddd8e189a525eb251f1 (table test-table [id=39431c34bef44daeb84ddde0b0e62a0c]) successfully deleted
I20260928 16:43:33.632471 28782 catalog_manager.cc:4841] TS 851722a3ca344203a371b0da8b397480 (127.26.101.3:43165): tablet 04b2f09c8d604ddd8e189a525eb251f1 (table test-table [id=39431c34bef44daeb84ddde0b0e62a0c]) successfully deleted
I20260928 16:43:33.634435 29465 ts_tablet_manager.cc:1918] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:43:33.634559 29465 log.cc:1196] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/04b2f09c8d604ddd8e189a525eb251f1
I20260928 16:43:33.635107 29465 ts_tablet_manager.cc:1939] T 04b2f09c8d604ddd8e189a525eb251f1 P 5add1b4b69004a26b71d68b631804fc8: Deleting consensus metadata
I20260928 16:43:33.636173 28779 catalog_manager.cc:4841] TS 5add1b4b69004a26b71d68b631804fc8 (127.26.101.1:34331): tablet 04b2f09c8d604ddd8e189a525eb251f1 (table test-table [id=39431c34bef44daeb84ddde0b0e62a0c]) successfully deleted
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:33.648167 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.611457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.26.101.1:34331)
0928 16:43:33.619344 (+ 7887us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.619352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.622950 (+ 3598us) server_negotiation.cc:407] Connection header received
0928 16:43:33.623237 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.623243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.623338 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.623426 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.625540 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.627061 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.629468 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.630164 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.630354 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.641774 (+ 11420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.641797 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.641819 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.641863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.645084 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.645952 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.645960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.645963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.646129 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.646601 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.646606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.646608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.647022 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.647416 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.647679 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.647890 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7716,"thread_start_us":7519,"threads_started":1}
W20260928 16:43:33.648974 28998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55674
W20260928 16:43:33.649688 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:33.649896 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28866
I20260928 16:43:33.670789 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29008
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:33.687321 29470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.671552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.26.101.2:46165)
0928 16:43:33.672119 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.672124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.672141 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:43:33.672215 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.672219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.672289 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.672384 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.673679 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.674478 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.675921 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.676379 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.676505 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.680867 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.680887 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.680904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.680948 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.684201 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.685092 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.685101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.685103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.685272 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.685656 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.685660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.685663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.686123 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.686367 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.686770 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.686964 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":330,"threads_started":1}
W20260928 16:43:33.688174 29139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51728
W20260928 16:43:33.688901 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:33.689056 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29008
I20260928 16:43:33.710263 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29148
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[28749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:33.727380 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:33.711101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.26.101.3:43165)
0928 16:43:33.711618 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:33.711624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:33.711643 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:43:33.711705 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:33.711708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:33.711776 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:33.711870 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:33.713743 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.714544 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.715988 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:33.716456 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:33.716561 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:33.721121 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:33.721140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:33.721178 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:33.721225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:33.724223 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.725029 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.725037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.725039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.725214 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:33.725626 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:33.725630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:33.725633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:33.726015 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:33.726312 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:33.726644 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 16:43:33.726878 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
W20260928 16:43:33.728154 29279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50548
W20260928 16:43:33.728830 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29148: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:33.729017 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29148
I20260928 16:43:33.752388 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 28763
I20260928 16:43:33.753721 28835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:33.870790 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 28763
2026-09-28T16:43:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9164 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[29475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[29475](info): set up 2 sockets
Sep 28 16:43:33 dist-test-slave-t0tw krb5kdc[29475](info): commencing operation
krb5kdc: starting...
W20260928 16:43:34.886969 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:35 dist-test-slave-t0tw krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:35.086544 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39353
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:39353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:35.409618 29488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:35.409960 29488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:35.410104 29488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:35.417621 29488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:35.417776 29488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:35.417856 29488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:35.417932 29488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:35.417994 29488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:35.428135 29488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:39353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:35.429535 29488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:35.430863 29488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:35.442924 29493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:35.442852 29494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:35.443176 29496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:35.443490 29488 server_base.cc:1034] running on GCE node
I20260928 16:43:35.444541 29488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:35.446110 29488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:35.447324 29488 hybrid_clock.cc:648] HybridClock initialized: now 1790613815447277 us; error 41 us; skew 500 ppm
I20260928 16:43:35.447681 29488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:35 dist-test-slave-t0tw krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613815, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:35.450730 29488 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:35.452198 29488 webserver.cc:460] Webserver started at http://127.26.101.62:38731/ using document root <none> and password file <none>
I20260928 16:43:35.452888 29488 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:35.452986 29488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:35.453245 29488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:35.456288 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "86a3d56daee84e5881a311b40b132802"
format_stamp: "Formatted at 2026-09-28 16:43:35 on dist-test-slave-t0tw"
server_key: "d7e56140d964d1632fb3fe83078c9b1f"
server_key_iv: "3f26c6720a01eec0617a945cb04ba7e3"
server_key_version: "encryptionkey@0"
I20260928 16:43:35.457000 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "86a3d56daee84e5881a311b40b132802"
format_stamp: "Formatted at 2026-09-28 16:43:35 on dist-test-slave-t0tw"
server_key: "d7e56140d964d1632fb3fe83078c9b1f"
server_key_iv: "3f26c6720a01eec0617a945cb04ba7e3"
server_key_version: "encryptionkey@0"
I20260928 16:43:35.462337 29488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:35.466022 29503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:35.467587 29488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:35.467762 29488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "86a3d56daee84e5881a311b40b132802"
format_stamp: "Formatted at 2026-09-28 16:43:35 on dist-test-slave-t0tw"
server_key: "d7e56140d964d1632fb3fe83078c9b1f"
server_key_iv: "3f26c6720a01eec0617a945cb04ba7e3"
server_key_version: "encryptionkey@0"
I20260928 16:43:35.467959 29488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:35.483840 29488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:35.488029 29488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:35.488318 29488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:35.506989 29488 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:39353
I20260928 16:43:35.507018 29555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:39353 every 8 connection(s)
I20260928 16:43:35.508471 29488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:35.512534 29556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:35.515398 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29488
I20260928 16:43:35.515646 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:35.516424 27028 external_mini_cluster.cc:1428] Setting key fdcf4b6af34efb490599d4a92da6b135
I20260928 16:43:35.525074 29556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802: Bootstrap starting.
I20260928 16:43:35.528409 29556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:35.529505 29556 log.cc:824] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:43:35 dist-test-slave-t0tw krb5kdc[29475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:35.532655 29556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802: No bootstrap required, opened a new log
I20260928 16:43:35.539111 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:35.518708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.26.101.62:39353)
0928 16:43:35.519575 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:35.519589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:35.519642 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:43:35.521363 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:35.521387 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:35.521828 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:35.522227 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:35.523750 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.525033 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.526351 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:35.527118 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:35.527305 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:35.531562 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:35.531570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:35.531586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:35.531624 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:35.534808 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:35.535598 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:35.535605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:35.535611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:35.535802 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:35.536202 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:35.536206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:35.536211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:35.536778 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:35.537105 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:35.537529 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 16:43:35.537979 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":407,"threads_started":1}
I20260928 16:43:35.540838 29556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } }
I20260928 16:43:35.541312 29556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:35.541430 29556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a3d56daee84e5881a311b40b132802, State: Initialized, Role: FOLLOWER
I20260928 16:43:35.542078 29556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } }
I20260928 16:43:35.542367 29556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:35.542510 29556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:35.542649 29556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:35.544795 29556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } }
I20260928 16:43:35.545256 29556 leader_election.cc:302] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a3d56daee84e5881a311b40b132802; no voters:
I20260928 16:43:35.545815 29556 leader_election.cc:288] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:35.545930 29561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:35.546556 29561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [term 1 LEADER]: Becoming Leader. State: Replica: 86a3d56daee84e5881a311b40b132802, State: Running, Role: LEADER
I20260928 16:43:35.547202 29561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } }
I20260928 16:43:35.547986 29556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:35.550211 29563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a3d56daee84e5881a311b40b132802. Latest consensus state: current_term: 1 leader_uuid: "86a3d56daee84e5881a311b40b132802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } } }
I20260928 16:43:35.550627 29563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:35.550750 29562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a3d56daee84e5881a311b40b132802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a3d56daee84e5881a311b40b132802" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39353 } } }
I20260928 16:43:35.551322 29562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:35.560242 29570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:35.565073 29570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:35.573872 29570 catalog_manager.cc:1347] Generated new cluster ID: facb389349a742d79343d07672aceb1a
I20260928 16:43:35.574004 29570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:35.591216 29570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:35.592551 29570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:35.601176 29570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a3d56daee84e5881a311b40b132802: Generated new TSK 0
I20260928 16:43:35.602139 29570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:35.654570 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:35.980893 29582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:35.981232 29582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:35.981367 29582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:35.988795 29582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:35.988943 29582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:35.989107 29582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:35.999562 29582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:36.001080 29582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:36.002496 29582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:36.015960 29587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.016039 29588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.016001 29590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:36.017021 29582 server_base.cc:1034] running on GCE node
I20260928 16:43:36.017688 29582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:36.018713 29582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:36.019954 29582 hybrid_clock.cc:648] HybridClock initialized: now 1790613816019903 us; error 41 us; skew 500 ppm
I20260928 16:43:36.020314 29582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:36 dist-test-slave-t0tw krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:36.023689 29582 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:36.025336 29582 webserver.cc:460] Webserver started at http://127.26.101.1:36661/ using document root <none> and password file <none>
I20260928 16:43:36.026046 29582 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:36.026147 29582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:36.026412 29582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:36.029456 29582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "d08e8a4699fd498da61d59eb4deb3d00"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "fe09dab65e3331362d62ab2ccd8fdba3"
server_key_iv: "8e08f6df98ddc81ae4363995251d5384"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.030231 29582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "d08e8a4699fd498da61d59eb4deb3d00"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "fe09dab65e3331362d62ab2ccd8fdba3"
server_key_iv: "8e08f6df98ddc81ae4363995251d5384"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.035718 29582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 16:43:36.040267 29597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.041814 29582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:36.041968 29582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "d08e8a4699fd498da61d59eb4deb3d00"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "fe09dab65e3331362d62ab2ccd8fdba3"
server_key_iv: "8e08f6df98ddc81ae4363995251d5384"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.042172 29582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:36.072726 29582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:36.077147 29582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:36.077415 29582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:36.078436 29582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:36.079906 29582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:36.079986 29582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.080075 29582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:36.080125 29582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.111706 29582 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:45537
I20260928 16:43:36.111763 29710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:45537 every 8 connection(s)
I20260928 16:43:36.113116 29582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:36.114024 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29582
I20260928 16:43:36.114255 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:36.114929 27028 external_mini_cluster.cc:1428] Setting key d423f09c74191b1c07488106e7a5f189
Sep 28 16:43:36 dist-test-slave-t0tw krb5kdc[29475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:43:36.137334 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.115788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:39029 (local address 127.26.101.62:39353)
0928 16:43:36.116310 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:36.116316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:36.117516 (+ 1200us) server_negotiation.cc:407] Connection header received
0928 16:43:36.120182 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:36.120188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:36.120272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:36.120376 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:36.122667 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.123576 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.125384 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.125997 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.126134 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.130390 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:36.130397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:36.130400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:36.130442 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:36.133320 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.134473 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.134478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.134481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.134737 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.135310 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.135314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.135316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.135712 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:36.135867 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:36.136802 (+ 935us) server_negotiation.cc:299] Negotiation successful
0928 16:43:36.137061 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:43:36.138378 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.116489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39353 (local address 127.26.101.1:39029)
0928 16:43:36.117326 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:36.117378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:36.119813 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:36.120728 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:36.120742 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:36.121527 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:36.122412 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.122444 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.123840 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.123844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.125192 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.125208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.126267 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.126274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.126521 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.127305 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:36.127330 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:36.130053 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:36.133540 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.133552 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.133574 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.134293 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.134877 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.134883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.134887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.135163 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.135866 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:36.135874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:36.136306 (+ 432us) client_negotiation.cc:769] Sending connection context
0928 16:43:36.136572 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 16:43:36.137074 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
I20260928 16:43:36.140141 29711 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39353
I20260928 16:43:36.140467 29711 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:36.141299 29711 heartbeater.cc:511] Master 127.26.101.62:39353 requested a full tablet report, sending...
I20260928 16:43:36.143889 29520 ts_manager.cc:194] Registered new tserver with Master: d08e8a4699fd498da61d59eb4deb3d00 (127.26.101.1:45537)
I20260928 16:43:36.145972 29520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:39029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:36.163615 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:36.482375 29717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:36.482705 29717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:36.482833 29717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:36.490214 29717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:36.490347 29717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:36.490505 29717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:36.500504 29717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:36.501874 29717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:36.503322 29717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:36.517789 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.517763 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:36.518090 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:36.519048 29717 server_base.cc:1034] running on GCE node
I20260928 16:43:36.519793 29717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:36.520807 29717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:36.522017 29717 hybrid_clock.cc:648] HybridClock initialized: now 1790613816521951 us; error 65 us; skew 500 ppm
I20260928 16:43:36.522329 29717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:36 dist-test-slave-t0tw krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:36.525480 29717 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:36.526912 29717 webserver.cc:460] Webserver started at http://127.26.101.2:35461/ using document root <none> and password file <none>
I20260928 16:43:36.527688 29717 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:36.527818 29717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:36.528116 29717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:36.531173 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "e098b3f1d1114cc8b0b46611bb45aef0"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "a598aa4ce8b2f412f0d969b5728bf475"
server_key_iv: "688c8d1dbc4f33a75972d46e7addb527"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.531977 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "e098b3f1d1114cc8b0b46611bb45aef0"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "a598aa4ce8b2f412f0d969b5728bf475"
server_key_iv: "688c8d1dbc4f33a75972d46e7addb527"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.537313 29717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:36.541008 29732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.542438 29717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:36.542608 29717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "e098b3f1d1114cc8b0b46611bb45aef0"
format_stamp: "Formatted at 2026-09-28 16:43:36 on dist-test-slave-t0tw"
server_key: "a598aa4ce8b2f412f0d969b5728bf475"
server_key_iv: "688c8d1dbc4f33a75972d46e7addb527"
server_key_version: "encryptionkey@0"
I20260928 16:43:36.542929 29717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:36.558615 29717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:36.564004 29717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:36.564276 29717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:36.565322 29717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:36.566756 29717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:36.566836 29717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.566941 29717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:36.566989 29717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:36.596858 29717 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:38865
I20260928 16:43:36.596906 29845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:38865 every 8 connection(s)
I20260928 16:43:36.598138 29717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:36.601361 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29717
I20260928 16:43:36.601531 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:36.601940 27028 external_mini_cluster.cc:1428] Setting key 8fb28066c298de38daf3439f58a1de5f
Sep 28 16:43:36 dist-test-slave-t0tw krb5kdc[29475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613816, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:43:36.618933 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.600776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36075 (local address 127.26.101.62:39353)
0928 16:43:36.600956 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:36.600961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:36.602426 (+ 1465us) server_negotiation.cc:407] Connection header received
0928 16:43:36.604586 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:36.604591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:36.604677 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:36.604798 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:36.606308 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.607278 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.608865 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.609284 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.609396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.613162 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:36.613177 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:36.613181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:36.613215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:36.615880 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.616632 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.616637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.616639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.616772 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:36.617210 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:36.617214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:36.617216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:36.617501 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:36.617638 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:36.618360 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 16:43:36.618596 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:43:36.620021 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:36.601346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39353 (local address 127.26.101.2:36075)
0928 16:43:36.602244 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:36.602295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:36.604313 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:36.604990 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:36.604998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:36.605543 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:36.606126 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.606145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.607438 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.607442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.608703 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:36.608716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:36.609511 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:36.609519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:36.609762 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:36.610447 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:36.610469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:36.612929 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:36.616037 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.616047 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.616064 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.616505 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.616887 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:36.616892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:36.616896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:36.617098 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:36.617633 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:36.617641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:36.618000 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 16:43:36.618240 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 16:43:36.618803 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:43:36.621644 29846 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39353
I20260928 16:43:36.622020 29846 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:36.622934 29846 heartbeater.cc:511] Master 127.26.101.62:39353 requested a full tablet report, sending...
I20260928 16:43:36.624576 29520 ts_manager.cc:194] Registered new tserver with Master: e098b3f1d1114cc8b0b46611bb45aef0 (127.26.101.2:38865)
I20260928 16:43:36.625383 29520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:36075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:36.647408 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:36.968811 29851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:36.969139 29851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:36.969280 29851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:36.976847 29851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:36.977000 29851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:36.977161 29851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:36.987250 29851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:36.988569 29851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:36.989890 29851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:37.003051 29856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:37.003108 29857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:37.003142 29859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:37.003899 29851 server_base.cc:1034] running on GCE node
I20260928 16:43:37.004530 29851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:37.005582 29851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:37.006801 29851 hybrid_clock.cc:648] HybridClock initialized: now 1790613817006737 us; error 54 us; skew 500 ppm
I20260928 16:43:37.007179 29851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:37 dist-test-slave-t0tw krb5kdc[29475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:37.010430 29851 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:37.012099 29851 webserver.cc:460] Webserver started at http://127.26.101.3:37439/ using document root <none> and password file <none>
I20260928 16:43:37.012859 29851 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:37.012967 29851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:37.013233 29851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:37.016230 29851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "16e921a9be1345ea9946b39557147df5"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-t0tw"
server_key: "6b8eb193e2da962ae2a4cb054324a510"
server_key_iv: "def049ee3eee5467330af4811ddbf399"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.016974 29851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "16e921a9be1345ea9946b39557147df5"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-t0tw"
server_key: "6b8eb193e2da962ae2a4cb054324a510"
server_key_iv: "def049ee3eee5467330af4811ddbf399"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.022223 29851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:37.025807 29866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.027405 29851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:43:37.027576 29851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "16e921a9be1345ea9946b39557147df5"
format_stamp: "Formatted at 2026-09-28 16:43:37 on dist-test-slave-t0tw"
server_key: "6b8eb193e2da962ae2a4cb054324a510"
server_key_iv: "def049ee3eee5467330af4811ddbf399"
server_key_version: "encryptionkey@0"
I20260928 16:43:37.027772 29851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:37.044160 29851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:37.048969 29851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:37.049274 29851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:37.050274 29851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:37.051710 29851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:37.051807 29851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.051930 29851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:37.051985 29851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:37.083695 29851 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:46601
I20260928 16:43:37.083751 29979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:46601 every 8 connection(s)
I20260928 16:43:37.084998 29851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:37.085966 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 29851
I20260928 16:43:37.086185 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:37.086740 27028 external_mini_cluster.cc:1428] Setting key 41a49bb9c8f0bc00c88ee12f690e8f3a
Sep 28 16:43:37 dist-test-slave-t0tw krb5kdc[29475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613817, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:37.106967 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.088025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:59927 (local address 127.26.101.62:39353)
0928 16:43:37.088219 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.088225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.089521 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 16:43:37.091550 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.091555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.091676 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.091764 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:37.093375 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.094141 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.095830 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.096263 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.096378 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.100054 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:37.100059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:37.100063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:37.100101 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:37.103640 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.104485 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.104489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.104492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.104637 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:37.105161 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:37.105164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:37.105167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:37.105490 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:37.105606 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:37.106441 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 16:43:37.106639 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:37.108163 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:37.088584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39353 (local address 127.26.101.3:59927)
0928 16:43:37.089369 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:37.089415 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:37.091271 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:37.092024 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:37.092033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:37.092577 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:37.093192 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.093212 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.094301 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.094304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.095673 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:37.095687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:37.096503 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:37.096510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:37.096749 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:37.097421 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:37.097444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:37.099788 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:37.103823 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.103833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.103850 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.104363 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.104813 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:37.104816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:37.104819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:37.105036 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:37.105649 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:37.105657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:37.106023 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 16:43:37.106297 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 16:43:37.106898 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":295,"threads_started":1}
I20260928 16:43:37.109730 29980 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39353
I20260928 16:43:37.110049 29980 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:37.110809 29980 heartbeater.cc:511] Master 127.26.101.62:39353 requested a full tablet report, sending...
I20260928 16:43:37.112501 29520 ts_manager.cc:194] Registered new tserver with Master: 16e921a9be1345ea9946b39557147df5 (127.26.101.3:46601)
I20260928 16:43:37.113488 29520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:59927
I20260928 16:43:37.115841 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:43:37.148633 29713 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:37.145658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.26.101.62:39353)
0928 16:43:37.145847 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:37.145851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:37.146390 (+ 539us) server_negotiation.cc:407] Connection header received
0928 16:43:37.146605 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:37.146611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:37.146715 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:37.146843 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:37.148433 (+ 1590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41072: BlockingRecv error: recv got EOF from 127.0.0.1:41072 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:43:37.149672 29711 heartbeater.cc:503] Master 127.26.101.62:39353 was elected leader, sending a full tablet report...
I20260928 16:43:37.151688 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29582
W20260928 16:43:37.154735 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29582: Not authorized: Client connection negotiation failed: client connection to 127.26.101.1:45537: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:43:37.154945 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29582
I20260928 16:43:37.169220 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29717
W20260928 16:43:37.172196 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29717: Not authorized: Client connection negotiation failed: client connection to 127.26.101.2:38865: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:43:37.172403 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29717
I20260928 16:43:37.186702 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29851
W20260928 16:43:37.189740 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29851: Not authorized: Client connection negotiation failed: client connection to 127.26.101.3:46601: server requires authentication, but client does not have Kerberos credentials available
I20260928 16:43:37.189883 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29851
I20260928 16:43:37.204670 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 29488
I20260928 16:43:37.205938 29551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:37.310489 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 29488
2026-09-28T16:43:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3436 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:37 dist-test-slave-t0tw krb5kdc[29997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:37 dist-test-slave-t0tw krb5kdc[29997](info): set up 2 sockets
Sep 28 16:43:37 dist-test-slave-t0tw krb5kdc[29997](info): commencing operation
krb5kdc: starting...
W20260928 16:43:38.322907 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:38 dist-test-slave-t0tw krb5kdc[29997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:38.524583 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42335
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:42335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:38.858305 30010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:38.858647 30010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:38.858788 30010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:38.866430 30010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:38.866580 30010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:38.866671 30010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:38.866796 30010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:38.866861 30010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:38.877585 30010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:42335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:38.879055 30010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:38.880465 30010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:38.892627 30015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:38.892566 30018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.893748 30010 server_base.cc:1034] running on GCE node
W20260928 16:43:38.893927 30016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:38.895006 30010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:38.896962 30010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:38.898262 30010 hybrid_clock.cc:648] HybridClock initialized: now 1790613818898198 us; error 80 us; skew 500 ppm
I20260928 16:43:38.898695 30010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:38 dist-test-slave-t0tw krb5kdc[29997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613818, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:38.908751 30010 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:38.910568 30010 webserver.cc:460] Webserver started at http://127.26.101.62:37617/ using document root <none> and password file <none>
I20260928 16:43:38.911525 30010 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:38.911674 30010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:38.912002 30010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:38.915232 30010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "871e1f543a864effa0ea09150c6d2694"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-t0tw"
server_key: "c72300f895cf8571274f01bfb36efa75"
server_key_iv: "377104a01deee835c2ebc5797aabbbad"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.916043 30010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "871e1f543a864effa0ea09150c6d2694"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-t0tw"
server_key: "c72300f895cf8571274f01bfb36efa75"
server_key_iv: "377104a01deee835c2ebc5797aabbbad"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.922086 30010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 16:43:38.926142 30025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:38.927783 30010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:38.927963 30010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "871e1f543a864effa0ea09150c6d2694"
format_stamp: "Formatted at 2026-09-28 16:43:38 on dist-test-slave-t0tw"
server_key: "c72300f895cf8571274f01bfb36efa75"
server_key_iv: "377104a01deee835c2ebc5797aabbbad"
server_key_version: "encryptionkey@0"
I20260928 16:43:38.928150 30010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:38.960423 30010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:38.965246 30010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:38.965510 30010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:38.983994 30010 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:42335
I20260928 16:43:38.984045 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:42335 every 8 connection(s)
I20260928 16:43:38.985450 30010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:38.989355 30078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:38.995232 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30010
I20260928 16:43:38.995465 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:38.996165 27028 external_mini_cluster.cc:1428] Setting key ed092ad2bfe5af5b0d652b959944d05f
I20260928 16:43:39.002157 30078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694: Bootstrap starting.
I20260928 16:43:39.006276 30078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:39.007664 30078 log.cc:824] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:39.011008 30078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694: No bootstrap required, opened a new log
Sep 28 16:43:39 dist-test-slave-t0tw krb5kdc[29997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:39.021168 30078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } }
I20260928 16:43:39.021693 30078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:39.021807 30078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871e1f543a864effa0ea09150c6d2694, State: Initialized, Role: FOLLOWER
I20260928 16:43:39.022516 30078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } }
I20260928 16:43:39.022814 30078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:39.022989 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:38.998432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.26.101.62:42335)
0928 16:43:38.999392 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:38.999408 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:38.999470 (+ 62us) server_negotiation.cc:407] Connection header received
0928 16:43:39.001998 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:39.002033 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:39.002625 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:39.003205 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:39.004675 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.006657 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.008266 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.009108 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.009334 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.013282 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:39.013294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:39.013322 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:39.013383 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:39.017575 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:39.018367 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:39.018377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:39.018386 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:39.018632 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:39.019107 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:39.019112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:39.019116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:39.019958 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:39.020300 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:39.021032 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 16:43:39.021522 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":387,"threads_started":1}
I20260928 16:43:39.022967 30078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:39.023252 30078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:39.025740 30078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } }
I20260928 16:43:39.026290 30078 leader_election.cc:302] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871e1f543a864effa0ea09150c6d2694; no voters:
I20260928 16:43:39.026930 30078 leader_election.cc:288] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:39.027122 30083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:39.027359 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:39.028306 30083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [term 1 LEADER]: Becoming Leader. State: Replica: 871e1f543a864effa0ea09150c6d2694, State: Running, Role: LEADER
I20260928 16:43:39.028791 30083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } }
I20260928 16:43:39.029562 30078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:39.032116 30085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871e1f543a864effa0ea09150c6d2694. Latest consensus state: current_term: 1 leader_uuid: "871e1f543a864effa0ea09150c6d2694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } } }
I20260928 16:43:39.032545 30085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:39.033231 30084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871e1f543a864effa0ea09150c6d2694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e1f543a864effa0ea09150c6d2694" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42335 } } }
I20260928 16:43:39.033566 30084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:39.035645 30088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:39.041105 30088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:39.054384 30088 catalog_manager.cc:1347] Generated new cluster ID: 93118fdcfde248cfa1e49b073a9b2d68
I20260928 16:43:39.054602 30088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:39.071641 30088 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:39.073634 30088 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:39.082908 30088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871e1f543a864effa0ea09150c6d2694: Generated new TSK 0
I20260928 16:43:39.084692 30088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:39.104410 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:39.092060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.26.101.62:42335)
0928 16:43:39.092289 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:39.092296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:39.095756 (+ 3460us) server_negotiation.cc:407] Connection header received
0928 16:43:39.095908 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:39.095914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:39.096007 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:39.096152 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:39.097351 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.098596 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.100276 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:39.100995 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:39.101156 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:39.102469 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:39.102477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:39.102483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:39.103474 (+ 991us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:39.103561 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:39.103619 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:39.103751 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:39.103924 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 16:43:39.104044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:39.112442 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30010
I20260928 16:43:39.114584 30073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:39.206138 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30010
2026-09-28T16:43:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1890 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:39 dist-test-slave-t0tw krb5kdc[30112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:39 dist-test-slave-t0tw krb5kdc[30112](info): set up 2 sockets
Sep 28 16:43:39 dist-test-slave-t0tw krb5kdc[30112](info): commencing operation
krb5kdc: starting...
W20260928 16:43:40.218815 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:40 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:40.417116 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:36623
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:36623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:40.741935 30125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:40.742244 30125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:40.742372 30125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:40.749826 30125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:40.749950 30125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:40.749998 30125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:40.750082 30125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:40.750140 30125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:40.760515 30125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:36623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:36623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:40.761904 30125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:40.763280 30125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:40.775490 30130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:40.775532 30133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:40.775506 30131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:40.776582 30125 server_base.cc:1034] running on GCE node
I20260928 16:43:40.777400 30125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:40.778884 30125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:40.780133 30125 hybrid_clock.cc:648] HybridClock initialized: now 1790613820780052 us; error 49 us; skew 500 ppm
I20260928 16:43:40.780478 30125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:40 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:40.783720 30125 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:40.785313 30125 webserver.cc:460] Webserver started at http://127.26.101.62:40441/ using document root <none> and password file <none>
I20260928 16:43:40.785997 30125 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:40.786106 30125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:40.786406 30125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:40.789369 30125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "041d9850210c435081b1f71f3f31d7a4"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-t0tw"
server_key: "0d32184f7332c05fc7d4db347922d1cd"
server_key_iv: "06aa4ea6d62f2a3e56527ec79aea1c7d"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.790107 30125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "041d9850210c435081b1f71f3f31d7a4"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-t0tw"
server_key: "0d32184f7332c05fc7d4db347922d1cd"
server_key_iv: "06aa4ea6d62f2a3e56527ec79aea1c7d"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.795346 30125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 16:43:40.798791 30140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:40.800192 30125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:40.800369 30125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "041d9850210c435081b1f71f3f31d7a4"
format_stamp: "Formatted at 2026-09-28 16:43:40 on dist-test-slave-t0tw"
server_key: "0d32184f7332c05fc7d4db347922d1cd"
server_key_iv: "06aa4ea6d62f2a3e56527ec79aea1c7d"
server_key_version: "encryptionkey@0"
I20260928 16:43:40.800573 30125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:40.821530 30125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:40.825932 30125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:40.826211 30125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:40.843271 30125 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:36623
I20260928 16:43:40.843279 30192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:36623 every 8 connection(s)
I20260928 16:43:40.844630 30125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:40.846572 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30125
I20260928 16:43:40.846756 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:40.847406 27028 external_mini_cluster.cc:1428] Setting key 271832655918ea75edfef11e5308fbe7
I20260928 16:43:40.848536 30193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:43:40 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:40.862932 30193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4: Bootstrap starting.
I20260928 16:43:40.866578 30193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:40.867715 30193 log.cc:824] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:40.871140 30193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4: No bootstrap required, opened a new log
I20260928 16:43:40.871460 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:40.849294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.26.101.62:36623)
0928 16:43:40.850080 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:40.850096 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:40.850148 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:43:40.852059 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:40.852089 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:40.852646 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:40.853139 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:40.854423 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.856498 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.857814 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:40.858857 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:40.859044 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:40.863382 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:40.863391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:40.863409 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:40.863455 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:40.866645 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.867555 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.867562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.867570 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.867776 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:40.868263 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:40.868269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:40.868272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:40.869096 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:40.869378 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:40.869891 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 16:43:40.870308 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":333,"threads_started":1}
I20260928 16:43:40.879823 30193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } }
I20260928 16:43:40.880211 30193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:40.880321 30193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041d9850210c435081b1f71f3f31d7a4, State: Initialized, Role: FOLLOWER
I20260928 16:43:40.880903 30193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } }
I20260928 16:43:40.881114 30193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:40.881245 30193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:40.881400 30193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:40.884025 30193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } }
I20260928 16:43:40.884552 30193 leader_election.cc:302] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041d9850210c435081b1f71f3f31d7a4; no voters:
I20260928 16:43:40.885157 30193 leader_election.cc:288] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:40.885321 30198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:40.885958 30198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [term 1 LEADER]: Becoming Leader. State: Replica: 041d9850210c435081b1f71f3f31d7a4, State: Running, Role: LEADER
I20260928 16:43:40.886493 30198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } }
I20260928 16:43:40.887228 30193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:40.889148 30200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041d9850210c435081b1f71f3f31d7a4. Latest consensus state: current_term: 1 leader_uuid: "041d9850210c435081b1f71f3f31d7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } } }
I20260928 16:43:40.889554 30200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:40.891008 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041d9850210c435081b1f71f3f31d7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041d9850210c435081b1f71f3f31d7a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36623 } } }
I20260928 16:43:40.891474 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:40.891760 30204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:40.897349 30204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:40.906632 30204 catalog_manager.cc:1347] Generated new cluster ID: a8fefaba69c840db88f2549c532c4f86
I20260928 16:43:40.906769 30204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:40.943940 30204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:40.946063 30204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:40.957278 30204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041d9850210c435081b1f71f3f31d7a4: Generated new TSK 0
I20260928 16:43:40.958396 30204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:41.019035 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:41.343514 30219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:41.343840 30219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:41.343978 30219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:41.351176 30219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:41.351292 30219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:41.351447 30219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:41.361220 30219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:41.362525 30219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:41.363858 30219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:41.376904 30224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.376952 30227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.376896 30225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.377836 30219 server_base.cc:1034] running on GCE node
I20260928 16:43:41.378499 30219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:41.379539 30219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:41.380792 30219 hybrid_clock.cc:648] HybridClock initialized: now 1790613821380717 us; error 46 us; skew 500 ppm
I20260928 16:43:41.381138 30219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:41 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:41.384405 30219 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:41.385901 30219 webserver.cc:460] Webserver started at http://127.26.101.1:41829/ using document root <none> and password file <none>
I20260928 16:43:41.386628 30219 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:41.386751 30219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:41.387105 30219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:41.390070 30219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "d772bd10937040dfacb21feeb3c4e4fc"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "95998ec5b7145f5965666d5fe40dadd4"
server_key_iv: "c4c99298db5edc33e5091d684447b5d1"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.390882 30219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "d772bd10937040dfacb21feeb3c4e4fc"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "95998ec5b7145f5965666d5fe40dadd4"
server_key_iv: "c4c99298db5edc33e5091d684447b5d1"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.396296 30219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 16:43:41.399839 30234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.401217 30219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:41.401387 30219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "d772bd10937040dfacb21feeb3c4e4fc"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "95998ec5b7145f5965666d5fe40dadd4"
server_key_iv: "c4c99298db5edc33e5091d684447b5d1"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.401575 30219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:41.418059 30219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:41.422011 30219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:41.422297 30219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:41.423305 30219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:41.424718 30219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:41.424799 30219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.424926 30219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:41.424981 30219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.456458 30219 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:46299
I20260928 16:43:41.456498 30347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:46299 every 8 connection(s)
I20260928 16:43:41.457844 30219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:41.467886 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30219
I20260928 16:43:41.468102 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:41.468739 27028 external_mini_cluster.cc:1428] Setting key bfb3a4ef9d3e75734f4c4775ce2787fe
Sep 28 16:43:41 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:41.478683 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.460475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:35881 (local address 127.26.101.62:36623)
0928 16:43:41.460932 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.460937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.461931 (+ 994us) server_negotiation.cc:407] Connection header received
0928 16:43:41.463879 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.463884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.463969 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.464068 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.465681 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.466406 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.468117 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.468546 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.468676 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.472422 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.472430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.472435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:41.472476 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.475506 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.476258 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.476263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.476267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.476412 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.476919 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.476922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.476924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.477195 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.477307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.478216 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.478409 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260928 16:43:41.480031 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.460952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:36623 (local address 127.26.101.1:35881)
0928 16:43:41.461722 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.461771 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.463574 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.464344 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.464354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.464889 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:41.465484 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.465503 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.466570 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.466574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.467955 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.467969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.468789 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.468796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.469037 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.469745 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:41.469778 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:41.472183 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:41.475655 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.475663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.475677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.476124 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.476525 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.476531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.476534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.476809 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.477316 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:41.477324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:41.477786 (+ 462us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.478081 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.478648 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":291,"threads_started":1}
I20260928 16:43:41.481810 30348 heartbeater.cc:347] Connected to a master server at 127.26.101.62:36623
I20260928 16:43:41.482136 30348 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:41.482899 30348 heartbeater.cc:511] Master 127.26.101.62:36623 requested a full tablet report, sending...
I20260928 16:43:41.485121 30158 ts_manager.cc:194] Registered new tserver with Master: d772bd10937040dfacb21feeb3c4e4fc (127.26.101.1:46299)
I20260928 16:43:41.486541 30158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:35881
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:41.514346 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:41.833402 30354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:41.833729 30354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:41.833879 30354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:41.841156 30354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:41.841290 30354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:41.841442 30354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:41.851276 30354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:41.852595 30354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:41.853892 30354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:41.867113 30359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.867022 30362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:41.867060 30360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:41.867859 30354 server_base.cc:1034] running on GCE node
I20260928 16:43:41.868479 30354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:41.869446 30354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:41.870677 30354 hybrid_clock.cc:648] HybridClock initialized: now 1790613821870639 us; error 26 us; skew 500 ppm
I20260928 16:43:41.871014 30354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:41 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:41.874195 30354 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:41.875706 30354 webserver.cc:460] Webserver started at http://127.26.101.2:40759/ using document root <none> and password file <none>
I20260928 16:43:41.876396 30354 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:41.876500 30354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:41.876756 30354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:41.879681 30354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "e4b714cb6d0148bd8c99fdfcaf54d81d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "c2a6ed2d7366216d0b5eb4cdc89ea694"
server_key_iv: "61032161f676218ea3c6530bdf4c21ca"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.880419 30354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "e4b714cb6d0148bd8c99fdfcaf54d81d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "c2a6ed2d7366216d0b5eb4cdc89ea694"
server_key_iv: "61032161f676218ea3c6530bdf4c21ca"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.885573 30354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 16:43:41.889194 30369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.890578 30354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:41.890723 30354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "e4b714cb6d0148bd8c99fdfcaf54d81d"
format_stamp: "Formatted at 2026-09-28 16:43:41 on dist-test-slave-t0tw"
server_key: "c2a6ed2d7366216d0b5eb4cdc89ea694"
server_key_iv: "61032161f676218ea3c6530bdf4c21ca"
server_key_version: "encryptionkey@0"
I20260928 16:43:41.890893 30354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:41.913347 30354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:41.917840 30354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:41.918104 30354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:41.919127 30354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:41.920576 30354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:41.920655 30354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.920750 30354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:41.920794 30354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:41.952100 30354 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:36969
I20260928 16:43:41.952162 30482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:36969 every 8 connection(s)
I20260928 16:43:41.953440 30354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:41.963377 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30354
I20260928 16:43:41.963570 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:41.964092 27028 external_mini_cluster.cc:1428] Setting key e88cc707594c0b4721749ee7e2b48cbe
Sep 28 16:43:41 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613821, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:41.974264 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.956178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:56455 (local address 127.26.101.62:36623)
0928 16:43:41.956364 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:41.956369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:41.957546 (+ 1177us) server_negotiation.cc:407] Connection header received
0928 16:43:41.959480 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:41.959485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:41.959558 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:41.959652 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:41.961278 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.961982 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.963949 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.964418 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.964518 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.968418 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:41.968424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:41.968427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:41.968460 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:41.970965 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.971771 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.971775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.971777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.971907 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:41.972445 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:41.972450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:41.972454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:41.972807 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:41.972970 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:41.973690 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 16:43:41.973933 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:43:41.975172 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:41.956646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:36623 (local address 127.26.101.2:56455)
0928 16:43:41.957369 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:41.957407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:41.959158 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:41.959891 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:41.959900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:41.960469 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:41.961070 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.961088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.962144 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.962148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.963804 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:41.963821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:41.964659 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:41.964666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:41.964921 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:41.965660 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:41.965683 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:41.968176 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:41.971150 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.971160 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.971179 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.971629 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.972022 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:41.972027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:41.972031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:41.972293 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:41.972933 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:41.972939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:41.973293 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 16:43:41.973539 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 16:43:41.973960 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":270,"threads_started":1}
I20260928 16:43:41.976732 30484 heartbeater.cc:347] Connected to a master server at 127.26.101.62:36623
I20260928 16:43:41.977052 30484 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:41.977811 30484 heartbeater.cc:511] Master 127.26.101.62:36623 requested a full tablet report, sending...
I20260928 16:43:41.979388 30158 ts_manager.cc:194] Registered new tserver with Master: e4b714cb6d0148bd8c99fdfcaf54d81d (127.26.101.2:36969)
I20260928 16:43:41.980233 30158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:56455
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:42.010267 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:42.338718 30489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:42.339023 30489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:42.339195 30489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:42.346573 30489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:42.346711 30489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:42.346863 30489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:42.356902 30489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:42.358292 30489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:42.359658 30489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:42.373039 30495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.373297 30497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:42.373051 30494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:42.374395 30489 server_base.cc:1034] running on GCE node
I20260928 16:43:42.375056 30489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:42.376106 30489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:42.377355 30489 hybrid_clock.cc:648] HybridClock initialized: now 1790613822377290 us; error 38 us; skew 500 ppm
I20260928 16:43:42.377693 30489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:42.381291 30489 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:42.382798 30489 webserver.cc:460] Webserver started at http://127.26.101.3:40165/ using document root <none> and password file <none>
I20260928 16:43:42.383531 30489 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:42.383649 30489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:42.383919 30489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:42.386866 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "61af81a990fb4527b6e0452c63e6598e"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-t0tw"
server_key: "62429502eb6b2778f16854c9960e36d5"
server_key_iv: "eee2ae9f9bba07d8116694944faa5e67"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.387758 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "61af81a990fb4527b6e0452c63e6598e"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-t0tw"
server_key: "62429502eb6b2778f16854c9960e36d5"
server_key_iv: "eee2ae9f9bba07d8116694944faa5e67"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.393258 30489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:42.397050 30504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.398732 30489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:42.398896 30489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "61af81a990fb4527b6e0452c63e6598e"
format_stamp: "Formatted at 2026-09-28 16:43:42 on dist-test-slave-t0tw"
server_key: "62429502eb6b2778f16854c9960e36d5"
server_key_iv: "eee2ae9f9bba07d8116694944faa5e67"
server_key_version: "encryptionkey@0"
I20260928 16:43:42.399153 30489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:42.416440 30489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:42.420967 30489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:42.421243 30489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:42.422246 30489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:42.423732 30489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:42.423810 30489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.423906 30489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:42.423950 30489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:42.455156 30489 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:37223
I20260928 16:43:42.455183 30617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:37223 every 8 connection(s)
I20260928 16:43:42.456837 30489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:42.459503 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30489
I20260928 16:43:42.459707 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:42.460305 27028 external_mini_cluster.cc:1428] Setting key 4868bf28c1410d52db427ee3bc241cff
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:42.478386 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.459816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:39583 (local address 127.26.101.62:36623)
0928 16:43:42.460050 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.460056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.461360 (+ 1304us) server_negotiation.cc:407] Connection header received
0928 16:43:42.463760 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.463767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.463884 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.463983 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.465717 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.466493 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.468144 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.468571 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.468667 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.472204 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.472210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.472215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:42.472250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.475158 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.476079 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.476085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.476088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.476297 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.476768 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.476771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.476773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.477050 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.477187 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.477919 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.478119 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 16:43:42.479373 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.460295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:36623 (local address 127.26.101.3:39583)
0928 16:43:42.461201 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:42.461250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:42.463391 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:42.464190 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:42.464199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:42.464794 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:42.465489 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.465509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.466697 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.466701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.467993 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:42.468006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.468813 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.468819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:42.469021 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.469713 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:42.469736 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:42.471976 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:42.475297 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:42.475306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:42.475324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:42.475945 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:42.476440 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:42.476444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:42.476447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:42.476648 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:42.477169 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:42.477176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:42.477522 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 16:43:42.477788 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 16:43:42.478201 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":361,"threads_started":1}
I20260928 16:43:42.480835 30618 heartbeater.cc:347] Connected to a master server at 127.26.101.62:36623
I20260928 16:43:42.481129 30618 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:42.481879 30618 heartbeater.cc:511] Master 127.26.101.62:36623 requested a full tablet report, sending...
I20260928 16:43:42.483398 30158 ts_manager.cc:194] Registered new tserver with Master: 61af81a990fb4527b6e0452c63e6598e (127.26.101.3:37223)
I20260928 16:43:42.484282 30158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:39583
I20260928 16:43:42.488761 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:42.489732 30348 heartbeater.cc:503] Master 127.26.101.62:36623 was elected leader, sending a full tablet report...
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:42.537232 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.523483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.26.101.62:36623)
0928 16:43:42.523681 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.523686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.524167 (+ 481us) server_negotiation.cc:407] Connection header received
0928 16:43:42.524317 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.524321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.524387 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.524481 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.525906 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.526683 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.527969 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.528410 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.528509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.532342 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.532347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.532351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:42.532390 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.534837 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.535524 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.535528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.535530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.535677 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.535976 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.535979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.535981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.536331 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.536520 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.536802 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.536979 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 16:43:42.537817 30158 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58530
I20260928 16:43:42.539647 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30219
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:42.555974 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.540766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.26.101.1:46299)
0928 16:43:42.541372 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.541381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.541405 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:43:42.541461 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.541468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.541675 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.541820 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.543013 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.544078 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.545318 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.545910 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.546028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.550110 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.550124 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.550141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:42.550179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.553123 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.553852 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.553859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.553862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.553993 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.554369 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.554373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.554375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.554731 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.555025 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.555342 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.555609 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":284,"threads_started":1}
W20260928 16:43:42.557286 30343 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44146
W20260928 16:43:42.558485 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:42.558686 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30219
I20260928 16:43:42.573204 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30354
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:42.589401 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.574101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.26.101.2:36969)
0928 16:43:42.574619 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.574631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.574666 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:43:42.574749 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.574756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.574972 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.575172 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.576358 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.577405 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.578653 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.579289 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.579390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.583584 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.583598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.583614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:42.583653 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.586498 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.587396 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.587404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.587407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.587554 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.587996 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.588000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.588002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.588383 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.588634 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.588892 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.589105 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
W20260928 16:43:42.590770 30478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47246
W20260928 16:43:42.592002 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:42.592159 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30354
I20260928 16:43:42.607299 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30489
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:42.625087 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:42.608243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.26.101.3:37223)
0928 16:43:42.608872 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:42.608879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:42.608910 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:43:42.608970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:42.608976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:42.609200 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:42.609336 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:42.610770 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.611928 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.613342 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:42.614021 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:42.614142 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:42.618693 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:42.618708 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:42.618726 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:42.618790 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:42.621909 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.622697 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.622705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.622708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.622854 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:42.623424 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:42.623429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:42.623431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:42.623850 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:42.624110 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:42.624488 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 16:43:42.624714 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":303,"threads_started":1}
W20260928 16:43:42.626537 30613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52508
W20260928 16:43:42.627741 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:43:42.627923 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30489
I20260928 16:43:42.642076 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30125
I20260928 16:43:42.643419 30188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:42.747255 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30125
2026-09-28T16:43:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3545 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30635](info): set up 2 sockets
Sep 28 16:43:42 dist-test-slave-t0tw krb5kdc[30635](info): commencing operation
krb5kdc: starting...
W20260928 16:43:43.759443 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:43 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:43.966933 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40281
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:44.297765 30648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:44.298106 30648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:44.298240 30648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:44.305720 30648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:44.305857 30648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:44.305930 30648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:44.306007 30648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:44.306070 30648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:44.316248 30648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:44.317637 30648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:44.319130 30648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:44.331311 30653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.331284 30654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.331358 30656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:44.332043 30648 server_base.cc:1034] running on GCE node
I20260928 16:43:44.332875 30648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:44.334254 30648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:44.335471 30648 hybrid_clock.cc:648] HybridClock initialized: now 1790613824335402 us; error 50 us; skew 500 ppm
I20260928 16:43:44.335862 30648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:44 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:44.343953 30648 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:44.345546 30648 webserver.cc:460] Webserver started at http://127.26.101.62:34647/ using document root <none> and password file <none>
I20260928 16:43:44.346269 30648 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:44.346374 30648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:44.346692 30648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:44.349716 30648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "cc32415f31994f25b10b7133da928939"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "6c053577fa36a4e087a6c80cee64a380"
server_key_iv: "360f36d6c1be26f2d41dad7b7d09005c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.350481 30648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "cc32415f31994f25b10b7133da928939"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "6c053577fa36a4e087a6c80cee64a380"
server_key_iv: "360f36d6c1be26f2d41dad7b7d09005c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.355717 30648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 16:43:44.359391 30663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:44.360901 30648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:44.361095 30648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "cc32415f31994f25b10b7133da928939"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "6c053577fa36a4e087a6c80cee64a380"
server_key_iv: "360f36d6c1be26f2d41dad7b7d09005c"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.361301 30648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:44.380764 30648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:44.385345 30648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:44.385630 30648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:44.403880 30648 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:40281
I20260928 16:43:44.403913 30715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:40281 every 8 connection(s)
I20260928 16:43:44.405297 30648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:44.406775 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30648
I20260928 16:43:44.407011 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:44.407815 27028 external_mini_cluster.cc:1428] Setting key 462f1f5dd01c8ecaad8ce226c44e89aa
I20260928 16:43:44.410322 30716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:43:44 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:44.423741 30716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939: Bootstrap starting.
I20260928 16:43:44.428119 30716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:44.429416 30716 log.cc:824] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:44.433151 30716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939: No bootstrap required, opened a new log
I20260928 16:43:44.433338 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:44.409956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.26.101.62:40281)
0928 16:43:44.410747 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:44.410760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:44.410802 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:44.412490 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:44.412513 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:44.412929 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:44.413305 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:44.415279 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.416756 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.418490 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:44.419379 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:44.419545 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:44.424303 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:44.424315 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:44.424337 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:44.424388 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:44.428335 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.429250 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.429260 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.429269 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.429500 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:44.430034 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:44.430040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:44.430047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:44.430696 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:44.430975 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:44.431431 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 16:43:44.431909 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":356,"threads_started":1}
I20260928 16:43:44.441110 30716 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } }
I20260928 16:43:44.441527 30716 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:44.441634 30716 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc32415f31994f25b10b7133da928939, State: Initialized, Role: FOLLOWER
I20260928 16:43:44.442281 30716 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } }
I20260928 16:43:44.442507 30716 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:44.442642 30716 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:44.442807 30716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:44.445068 30716 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } }
I20260928 16:43:44.445631 30716 leader_election.cc:302] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc32415f31994f25b10b7133da928939; no voters:
I20260928 16:43:44.446484 30716 leader_election.cc:288] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:44.446785 30721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:44.447775 30721 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [term 1 LEADER]: Becoming Leader. State: Replica: cc32415f31994f25b10b7133da928939, State: Running, Role: LEADER
I20260928 16:43:44.448350 30721 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } }
I20260928 16:43:44.448846 30716 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:44.451254 30723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc32415f31994f25b10b7133da928939. Latest consensus state: current_term: 1 leader_uuid: "cc32415f31994f25b10b7133da928939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } } }
I20260928 16:43:44.451576 30723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:44.452133 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc32415f31994f25b10b7133da928939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32415f31994f25b10b7133da928939" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40281 } } }
I20260928 16:43:44.452423 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:44.454205 30727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:44.460121 30727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:44.471020 30727 catalog_manager.cc:1347] Generated new cluster ID: e7da9dd01ad94e3386feefa8bc54f29f
I20260928 16:43:44.471185 30727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:44.492033 30727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:44.493471 30727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:44.503021 30727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc32415f31994f25b10b7133da928939: Generated new TSK 0
I20260928 16:43:44.504108 30727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:44.558676 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:44.900020 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:44.900343 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:44.900476 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:44.908093 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:44.908221 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:44.908376 30743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:44.918640 30743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:44.920171 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:44.921566 30743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:44.935762 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.935577 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:44.935727 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:44.937011 30743 server_base.cc:1034] running on GCE node
I20260928 16:43:44.937772 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:44.938800 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:44.940014 30743 hybrid_clock.cc:648] HybridClock initialized: now 1790613824939943 us; error 67 us; skew 500 ppm
I20260928 16:43:44.940353 30743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:44 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:44.943779 30743 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:44.945307 30743 webserver.cc:460] Webserver started at http://127.26.101.1:35825/ using document root <none> and password file <none>
I20260928 16:43:44.946000 30743 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:44.946128 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:44.946408 30743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:44.949514 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "2d155b2f18d444aa884440ef78574e25"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "952478ca159bb14558459f38e7fa8375"
server_key_iv: "f317142696ba771fdd5ad1d1d6083746"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.950321 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "2d155b2f18d444aa884440ef78574e25"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "952478ca159bb14558459f38e7fa8375"
server_key_iv: "f317142696ba771fdd5ad1d1d6083746"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.956346 30743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 16:43:44.960292 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:44.961961 30743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:43:44.962147 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "2d155b2f18d444aa884440ef78574e25"
format_stamp: "Formatted at 2026-09-28 16:43:44 on dist-test-slave-t0tw"
server_key: "952478ca159bb14558459f38e7fa8375"
server_key_iv: "f317142696ba771fdd5ad1d1d6083746"
server_key_version: "encryptionkey@0"
I20260928 16:43:44.962383 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:44.993163 30743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:44.997790 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:44.998088 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:44.999204 30743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:45.000700 30743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:45.000779 30743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.000881 30743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:45.000948 30743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.031941 30743 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:36269
I20260928 16:43:45.031999 30872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:36269 every 8 connection(s)
I20260928 16:43:45.033231 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:45.039548 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30743
I20260928 16:43:45.039765 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:45.040395 27028 external_mini_cluster.cc:1428] Setting key bf0e52e03fb19b6f726fb512cdd0a95f
Sep 28 16:43:45 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613824, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:45.053948 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.035749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:56609 (local address 127.26.101.62:40281)
0928 16:43:45.036172 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:45.036176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:45.037081 (+ 905us) server_negotiation.cc:407] Connection header received
0928 16:43:45.038935 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:45.038941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:45.039024 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:45.039183 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:45.040817 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.041671 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.043468 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.043873 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.043992 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.047942 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:45.047947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:45.047951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:45.047989 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:45.050707 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:45.051554 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:45.051558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:45.051560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:45.051689 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:45.052160 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:45.052164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:45.052166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:45.052436 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:45.052585 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:45.053323 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 16:43:45.053573 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
I20260928 16:43:45.055061 30876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.036236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40281 (local address 127.26.101.1:56609)
0928 16:43:45.036925 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:45.036966 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:45.038653 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:45.039424 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:45.039433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:45.040013 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:45.040635 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:45.040657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.041851 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.041855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:45.043318 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:45.043333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.044112 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.044120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:45.044377 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.045067 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:45.045090 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:45.047603 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:45.050864 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:45.050875 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:45.050894 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:45.051412 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:45.051808 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:45.051813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:45.051817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:45.052035 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:45.052572 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:45.052581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:45.052939 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 16:43:45.053197 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 16:43:45.053771 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260928 16:43:45.056783 30873 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40281
I20260928 16:43:45.057085 30873 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:43:45.057818 30873 heartbeater.cc:511] Master 127.26.101.62:40281 requested a full tablet report, sending...
I20260928 16:43:45.060153 30680 ts_manager.cc:194] Registered new tserver with Master: 2d155b2f18d444aa884440ef78574e25 (127.26.101.1:36269)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:43:45.061874 30680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:56609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:45.087421 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:45.411428 30879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:45.411752 30879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:45.411903 30879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:45.419358 30879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:45.419487 30879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:45.419652 30879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:45.429896 30879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:45.431280 30879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:45.432632 30879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:45.447597 30885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.447736 30887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.447625 30884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:45.448607 30879 server_base.cc:1034] running on GCE node
I20260928 16:43:45.449321 30879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:45.450448 30879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:45.451782 30879 hybrid_clock.cc:648] HybridClock initialized: now 1790613825451726 us; error 44 us; skew 500 ppm
I20260928 16:43:45.452131 30879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:45 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:45.455826 30879 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:45.457442 30879 webserver.cc:460] Webserver started at http://127.26.101.2:42165/ using document root <none> and password file <none>
I20260928 16:43:45.458210 30879 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:45.458339 30879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:45.458669 30879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:45.461977 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "c71e6e93258a4f60b7857780643e4f5b"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "4fd142ebdf127e411e5e2bf649d832cd"
server_key_iv: "fadd58a01d03fbd17170a824fba97c58"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.462824 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "c71e6e93258a4f60b7857780643e4f5b"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "4fd142ebdf127e411e5e2bf649d832cd"
server_key_iv: "fadd58a01d03fbd17170a824fba97c58"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.468983 30879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:43:45.473060 30894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.474664 30879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 16:43:45.474861 30879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "c71e6e93258a4f60b7857780643e4f5b"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "4fd142ebdf127e411e5e2bf649d832cd"
server_key_iv: "fadd58a01d03fbd17170a824fba97c58"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.475117 30879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:45.501879 30879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:45.506592 30879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:45.506906 30879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:45.507961 30879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:45.509393 30879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:45.509477 30879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.509596 30879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:45.509663 30879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.541499 30879 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:39241
I20260928 16:43:45.541554 31007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:39241 every 8 connection(s)
I20260928 16:43:45.542811 30879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:45.547236 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 30879
I20260928 16:43:45.547430 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:45.547849 27028 external_mini_cluster.cc:1428] Setting key 65fb68c1f538546b347401dc63f218e7
Sep 28 16:43:45 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:45.565124 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.545494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:47231 (local address 127.26.101.62:40281)
0928 16:43:45.545729 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:45.545733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:45.546852 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 16:43:45.549080 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:45.549086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:45.549178 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:45.549270 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:45.550988 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.551747 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.553363 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.553818 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.553931 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.558234 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:45.558242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:45.558248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:45.558296 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:45.561535 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:45.562370 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:45.562376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:45.562380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:45.562560 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:45.563153 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:45.563158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:45.563162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:45.563543 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:45.563755 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:45.564488 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 16:43:45.564766 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 16:43:45.566277 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:45.545951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40281 (local address 127.26.101.2:47231)
0928 16:43:45.546671 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:45.546718 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:45.548775 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:45.549565 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:45.549575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:45.550155 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:45.550806 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:45.550826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.551910 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.551913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:45.553204 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:45.553222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:45.554055 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:45.554062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:45.554303 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:45.555054 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:45.555116 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:45.557842 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:45.561721 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:45.561729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:45.561747 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:45.562234 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:45.562697 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:45.562703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:45.562707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:45.563002 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:45.563703 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:45.563710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:45.564092 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 16:43:45.564344 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 16:43:45.564931 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:43:45.568116 31008 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40281
I20260928 16:43:45.568449 31008 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:45.569253 31008 heartbeater.cc:511] Master 127.26.101.62:40281 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:43:45.570956 30680 ts_manager.cc:194] Registered new tserver with Master: c71e6e93258a4f60b7857780643e4f5b (127.26.101.2:39241)
I20260928 16:43:45.572081 30680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:47231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:45.594846 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:45.923205 31013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:45.923529 31013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:45.923748 31013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:45.931746 31013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:45.931896 31013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:45.932063 31013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:45.942353 31013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:45.943822 31013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:45.945243 31013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:45.959343 31018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.959461 31021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:45.959316 31019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:45.960183 31013 server_base.cc:1034] running on GCE node
I20260928 16:43:45.960827 31013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:45.961784 31013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:45.963030 31013 hybrid_clock.cc:648] HybridClock initialized: now 1790613825962970 us; error 56 us; skew 500 ppm
I20260928 16:43:45.963536 31013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:45 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:45.967027 31013 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:45.968797 31013 webserver.cc:460] Webserver started at http://127.26.101.3:44177/ using document root <none> and password file <none>
I20260928 16:43:45.969542 31013 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:45.969660 31013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:45.969971 31013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:45.973224 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "9ed6c3d5de314c508d992bacba30f4c1"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "da01faa953594a1c8263c555a2a8eef4"
server_key_iv: "fa6a74335e6d37ca065c8e329e576c53"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.974042 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "9ed6c3d5de314c508d992bacba30f4c1"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "da01faa953594a1c8263c555a2a8eef4"
server_key_iv: "fa6a74335e6d37ca065c8e329e576c53"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.979918 31013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:43:45.984055 31028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:45.985802 31013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:43:45.985996 31013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "9ed6c3d5de314c508d992bacba30f4c1"
format_stamp: "Formatted at 2026-09-28 16:43:45 on dist-test-slave-t0tw"
server_key: "da01faa953594a1c8263c555a2a8eef4"
server_key_iv: "fa6a74335e6d37ca065c8e329e576c53"
server_key_version: "encryptionkey@0"
I20260928 16:43:45.986231 31013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:46.011581 31013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:46.016450 31013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:46.016767 31013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:46.017824 31013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:46.019284 31013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:46.019359 31013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:46.019482 31013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:46.019531 31013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:46.051456 31013 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:37993
I20260928 16:43:46.051524 31141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:37993 every 8 connection(s)
I20260928 16:43:46.052795 31013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:46.054527 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31013
I20260928 16:43:46.054706 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:46.055222 27028 external_mini_cluster.cc:1428] Setting key f02bd08379736036a849ef7f8882c4de
I20260928 16:43:46.065543 30873 heartbeater.cc:503] Master 127.26.101.62:40281 was elected leader, sending a full tablet report...
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613825, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:46.074162 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.055558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:46425 (local address 127.26.101.62:40281)
0928 16:43:46.055786 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.055791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.057224 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 16:43:46.059225 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.059229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.059329 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.059417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.060968 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.061802 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.063432 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.063886 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.063984 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.068134 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.068141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.068144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.068176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.070984 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.071804 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.071808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.071811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.071957 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.072440 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.072443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.072445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.072740 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.072858 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.073680 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.073884 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:43:46.075374 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.056150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40281 (local address 127.26.101.3:46425)
0928 16:43:46.057051 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:46.057112 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:46.058937 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:46.059627 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:46.059635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:46.060182 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:46.060791 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:46.060809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.061957 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.061961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:46.063296 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:46.063309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.064077 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.064082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:46.064304 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.065118 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:46.065162 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:46.067923 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:46.071176 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:46.071183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:46.071199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:46.071654 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:46.072101 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:46.072105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:46.072108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:46.072332 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:46.072896 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:46.072902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:46.073281 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 16:43:46.073532 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 16:43:46.073989 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":346,"threads_started":1}
I20260928 16:43:46.077127 31142 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40281
I20260928 16:43:46.077487 31142 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:46.078307 31142 heartbeater.cc:511] Master 127.26.101.62:40281 requested a full tablet report, sending...
I20260928 16:43:46.079861 30680 ts_manager.cc:194] Registered new tserver with Master: 9ed6c3d5de314c508d992bacba30f4c1 (127.26.101.3:37993)
I20260928 16:43:46.080874 30680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:46425
I20260928 16:43:46.083015 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:46.132632 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.115972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.26.101.1:36269)
0928 16:43:46.116661 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.116668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.116700 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:43:46.116771 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.116779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.117018 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.117151 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.118693 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.119869 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.121220 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.121812 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.121926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.126505 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.126523 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.126540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.126584 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.129551 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.130345 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.130353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.130356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.130497 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.130913 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.130917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.130919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.131351 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.131701 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.131983 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.132269 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":364,"threads_started":1}
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:46.151978 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.137144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.26.101.62:40281)
0928 16:43:46.137376 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.137382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.137548 (+ 166us) server_negotiation.cc:407] Connection header received
0928 16:43:46.137726 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.137732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.137804 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.137891 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.139219 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.139959 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.141283 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.141782 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.141898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.146312 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.146318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.146322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.146367 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.149050 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.149876 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.149881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.149883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.150030 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.150453 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.150457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.150459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.150834 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.151048 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.151487 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.151730 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 16:43:46.152618 30680 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156
I20260928 16:43:46.153831 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30743
I20260928 16:43:46.170908 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.154932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.26.101.1:36269)
0928 16:43:46.155169 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.155175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.155361 (+ 186us) server_negotiation.cc:407] Connection header received
0928 16:43:46.155549 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.155555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.155717 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.155824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.157474 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.158505 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.160301 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.160980 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.161078 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.164272 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.164279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.164283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.164334 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.167902 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.168844 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.168851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.168854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.169014 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.169451 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.169457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.169461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.169803 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.170048 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.170381 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.170599 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:43:46.171545 30868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:46.254240 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30743
I20260928 16:43:46.269470 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30879
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:46.287153 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.270427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.26.101.2:39241)
0928 16:43:46.271056 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.271064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.271135 (+ 71us) server_negotiation.cc:407] Connection header received
0928 16:43:46.271202 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.271209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.271428 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.271623 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.272914 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.274090 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.275746 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.276353 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.276482 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.281145 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.281160 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.281176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.281223 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.284367 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.285119 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.285146 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.285148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.285281 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.285609 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.285612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.285614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.285992 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.286245 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.286484 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.286724 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":327,"threads_started":1}
I20260928 16:43:46.288467 31003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:46.371590 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30879
I20260928 16:43:46.386796 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31013
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[30635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:46.404626 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:46.387833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.26.101.3:37993)
0928 16:43:46.388504 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:46.388512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:46.388543 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:43:46.388680 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:46.388687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:46.388908 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:46.389058 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:46.390373 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.391899 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.393246 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:46.393824 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:46.393943 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:46.398526 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:46.398539 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:46.398555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:46.398606 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:46.401568 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.402313 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.402321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.402323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.402481 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:46.402851 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:46.402855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:46.402857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:46.403306 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:46.403639 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:46.403949 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 16:43:46.404227 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":367,"threads_started":1}
I20260928 16:43:46.406097 31137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:46.488404 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31013
I20260928 16:43:46.502935 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 30648
I20260928 16:43:46.504310 30710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:46.609375 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 30648
2026-09-28T16:43:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3862 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[31161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[31161](info): set up 2 sockets
Sep 28 16:43:46 dist-test-slave-t0tw krb5kdc[31161](info): commencing operation
krb5kdc: starting...
W20260928 16:43:47.627590 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:47 dist-test-slave-t0tw krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:47.834846 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:35147
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:48.169296 31174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:48.169631 31174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:48.169790 31174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:48.177331 31174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:48.177479 31174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:48.177565 31174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:48.177649 31174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:48.177711 31174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:48.188262 31174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:35147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:48.189697 31174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:48.191143 31174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:48.203766 31182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:48.203892 31179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:48.203791 31180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:48.204668 31174 server_base.cc:1034] running on GCE node
I20260928 16:43:48.205560 31174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:48.207166 31174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:48.208384 31174 hybrid_clock.cc:648] HybridClock initialized: now 1790613828208332 us; error 51 us; skew 500 ppm
I20260928 16:43:48.208770 31174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:48 dist-test-slave-t0tw krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:48.212152 31174 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:48.213457 31174 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:48.213564 31174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:48.213897 31174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:48.217100 31174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "da7be27a1d13437a98c603533cf8b6ee"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "3adf71312cb6fca739c3c8b73a3ef86e"
server_key_iv: "b9050ea561e80d7101227e653e657a39"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.217897 31174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "da7be27a1d13437a98c603533cf8b6ee"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "3adf71312cb6fca739c3c8b73a3ef86e"
server_key_iv: "b9050ea561e80d7101227e653e657a39"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.223462 31174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 16:43:48.227710 31189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:48.229250 31174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:43:48.229437 31174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "da7be27a1d13437a98c603533cf8b6ee"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "3adf71312cb6fca739c3c8b73a3ef86e"
server_key_iv: "b9050ea561e80d7101227e653e657a39"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.229653 31174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:48.243551 31174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:48.248379 31174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:48.248643 31174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:48.266855 31174 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:35147
I20260928 16:43:48.266839 31240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:35147 every 8 connection(s)
I20260928 16:43:48.267933 31174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:48.272133 31241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:48.274922 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31174
I20260928 16:43:48.275203 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:48.275876 27028 external_mini_cluster.cc:1428] Setting key 10f55b1b069cd68d13e9e29d1014d244
I20260928 16:43:48.285261 31241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: Bootstrap starting.
I20260928 16:43:48.289458 31241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:48.290928 31241 log.cc:824] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:43:48 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:48.294361 31241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: No bootstrap required, opened a new log
I20260928 16:43:48.301115 31244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:48.278127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.26.101.62:35147)
0928 16:43:48.279002 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:48.279015 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:48.279060 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:43:48.280876 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:48.280903 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:48.281354 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:48.281744 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:48.283705 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.285052 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.286947 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.287852 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.288001 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:48.293105 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:48.293115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:48.293136 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:48.293178 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:48.296368 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.297296 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.297306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.297313 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.297520 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.297986 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.297991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.297993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.298666 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:48.298934 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:48.299476 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 16:43:48.299916 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":374,"threads_started":1}
I20260928 16:43:48.303198 31241 raft_consensus.cc:348] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } }
I20260928 16:43:48.303615 31241 raft_consensus.cc:374] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:48.303719 31241 raft_consensus.cc:729] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7be27a1d13437a98c603533cf8b6ee, State: Initialized, Role: FOLLOWER
I20260928 16:43:48.304333 31241 consensus_queue.cc:260] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } }
I20260928 16:43:48.304575 31241 raft_consensus.cc:388] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:48.304701 31241 raft_consensus.cc:482] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:48.304852 31241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:48.307070 31241 raft_consensus.cc:504] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } }
I20260928 16:43:48.307665 31241 leader_election.cc:302] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da7be27a1d13437a98c603533cf8b6ee; no voters:
I20260928 16:43:48.308323 31241 leader_election.cc:288] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:48.308452 31246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:48.309623 31246 raft_consensus.cc:686] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [term 1 LEADER]: Becoming Leader. State: Replica: da7be27a1d13437a98c603533cf8b6ee, State: Running, Role: LEADER
I20260928 16:43:48.310128 31246 consensus_queue.cc:237] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } }
I20260928 16:43:48.310714 31241 sys_catalog.cc:564] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:48.313172 31247 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da7be27a1d13437a98c603533cf8b6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } } }
I20260928 16:43:48.313584 31247 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:48.313484 31248 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7be27a1d13437a98c603533cf8b6ee. Latest consensus state: current_term: 1 leader_uuid: "da7be27a1d13437a98c603533cf8b6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7be27a1d13437a98c603533cf8b6ee" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35147 } } }
I20260928 16:43:48.313796 31248 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:48.315362 31255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:48.321398 31255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:48.330991 31255 catalog_manager.cc:1347] Generated new cluster ID: 9a7cb12165aa4c9b84db1c13b05aafec
I20260928 16:43:48.331169 31255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:48.362610 31255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:48.364076 31255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:48.373086 31255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: Generated new TSK 0
I20260928 16:43:48.374202 31255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:48.431627 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:48.759898 31267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:48.760201 31267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:48.760300 31267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:48.767905 31267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:48.768049 31267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:48.768167 31267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:48.778108 31267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:48.779541 31267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:48.780890 31267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:48.793771 31272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:48.793736 31273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:48.793795 31275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:48.794644 31267 server_base.cc:1034] running on GCE node
I20260928 16:43:48.795333 31267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:48.796366 31267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:48.797605 31267 hybrid_clock.cc:648] HybridClock initialized: now 1790613828797545 us; error 49 us; skew 500 ppm
I20260928 16:43:48.797989 31267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:48 dist-test-slave-t0tw krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:48.801308 31267 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:48.802551 31267 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:48.802662 31267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:48.803043 31267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:48.806399 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "be97fd95353d4337838095bfbcfd0d91"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "903a9a2fe687de532246e4bec4fac611"
server_key_iv: "0db64ab91305ba6ab91e22438b9b2f13"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.807276 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "be97fd95353d4337838095bfbcfd0d91"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "903a9a2fe687de532246e4bec4fac611"
server_key_iv: "0db64ab91305ba6ab91e22438b9b2f13"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.812569 31267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:43:48.816239 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:48.817575 31267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:43:48.817765 31267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "be97fd95353d4337838095bfbcfd0d91"
format_stamp: "Formatted at 2026-09-28 16:43:48 on dist-test-slave-t0tw"
server_key: "903a9a2fe687de532246e4bec4fac611"
server_key_iv: "0db64ab91305ba6ab91e22438b9b2f13"
server_key_version: "encryptionkey@0"
I20260928 16:43:48.817961 31267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:48.850647 31267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:48.855201 31267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:48.855437 31267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:48.856379 31267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:48.857813 31267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:48.857900 31267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:48.857983 31267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:48.858035 31267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:48.888834 31267 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:42181
I20260928 16:43:48.888916 31394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:42181 every 8 connection(s)
I20260928 16:43:48.889704 31267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:48.892717 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31267
I20260928 16:43:48.892910 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:48.893705 27028 external_mini_cluster.cc:1428] Setting key ba10b005ccadf479086cce94eed0ec3b
Sep 28 16:43:48 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613828, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:48.912321 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:48.892197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:48211 (local address 127.26.101.62:35147)
0928 16:43:48.892768 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:48.892775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:48.894343 (+ 1568us) server_negotiation.cc:407] Connection header received
0928 16:43:48.895834 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:48.895839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:48.895912 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:48.895995 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:48.898080 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.898854 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.901044 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.901473 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.901580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:48.905693 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:48.905699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:48.905702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:48.905742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:48.908420 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.909541 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.909547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.909550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.909707 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:48.910265 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:48.910269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:48.910271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:48.910614 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:48.910783 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:48.911839 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0928 16:43:48.912043 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:43:48.913859 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:48.892746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35147 (local address 127.26.101.1:48211)
0928 16:43:48.893625 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:48.893689 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:48.895555 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:48.896279 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:48.896291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:48.897019 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:48.897870 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:48.897896 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.899024 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.899029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:48.900903 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:48.900919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:48.901700 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:48.901707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:48.901963 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:48.902728 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:48.902754 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:48.905370 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:48.908604 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:48.908615 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:48.908636 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:48.909332 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:48.909829 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:48.909835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:48.909838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:48.910126 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:48.910754 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:48.910764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:48.911325 (+ 561us) client_negotiation.cc:769] Sending connection context
0928 16:43:48.911700 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 16:43:48.912325 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260928 16:43:48.915550 31395 heartbeater.cc:347] Connected to a master server at 127.26.101.62:35147
I20260928 16:43:48.915904 31395 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:43:48.916651 31395 heartbeater.cc:511] Master 127.26.101.62:35147 requested a full tablet report, sending...
I20260928 16:43:48.918944 31206 ts_manager.cc:194] Registered new tserver with Master: be97fd95353d4337838095bfbcfd0d91 (127.26.101.1:42181)
I20260928 16:43:48.920535 31206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:48211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:48.943701 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:49.278232 31401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:49.278543 31401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:49.278692 31401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:49.286054 31401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:49.286187 31401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:49.286351 31401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:49.296386 31401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:49.297732 31401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:49.299065 31401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:49.312338 31407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:49.312371 31409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:49.312420 31406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:49.313344 31401 server_base.cc:1034] running on GCE node
I20260928 16:43:49.313964 31401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:49.314909 31401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:49.316143 31401 hybrid_clock.cc:648] HybridClock initialized: now 1790613829316091 us; error 62 us; skew 500 ppm
I20260928 16:43:49.316468 31401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:49 dist-test-slave-t0tw krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:49.319598 31401 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:43:49.320780 31401 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:49.320873 31401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:49.321182 31401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:49.324190 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "1c061e5b828c483ba4d13df03389f50d"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "02a1fe68430ec8628ab24fbf2f6158c8"
server_key_iv: "079ce944ace22fefa78e9f124ab15d2b"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.324995 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "1c061e5b828c483ba4d13df03389f50d"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "02a1fe68430ec8628ab24fbf2f6158c8"
server_key_iv: "079ce944ace22fefa78e9f124ab15d2b"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.330010 31401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:49.333783 31415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.335250 31401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:49.335389 31401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "1c061e5b828c483ba4d13df03389f50d"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "02a1fe68430ec8628ab24fbf2f6158c8"
server_key_iv: "079ce944ace22fefa78e9f124ab15d2b"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.335594 31401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:49.360039 31401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:49.364564 31401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:49.364845 31401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:49.365857 31401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:49.367293 31401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:49.367385 31401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.367491 31401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:49.367560 31401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.398572 31401 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:35769
I20260928 16:43:49.398620 31528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:35769 every 8 connection(s)
I20260928 16:43:49.399471 31401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:49.403792 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31401
I20260928 16:43:49.403988 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:49.404417 27028 external_mini_cluster.cc:1428] Setting key 288bd4426924e248a0986595054b72e2
Sep 28 16:43:49 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:49.420442 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.402192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:45723 (local address 127.26.101.62:35147)
0928 16:43:49.402386 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.402391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.403561 (+ 1170us) server_negotiation.cc:407] Connection header received
0928 16:43:49.405544 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.405550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.405637 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.405734 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:49.407406 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.408100 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.409660 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.410105 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.410221 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.414391 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:49.414398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:49.414402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:49.414443 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:49.417109 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.417980 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.417983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.417986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.418130 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.418614 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.418618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.418621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.418922 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:49.419035 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.420002 (+ 967us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.420205 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:49.422049 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.402680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35147 (local address 127.26.101.2:45723)
0928 16:43:49.403410 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:49.403448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:49.405243 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:49.405974 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:49.405983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:49.406594 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:49.407232 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.407258 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.408258 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.408261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.409515 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.409528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.410332 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.410340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.410558 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.411353 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:49.411383 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:49.414126 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:49.417258 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:49.417267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:49.417285 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:49.417863 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:49.418236 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:49.418241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:49.418245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:49.418491 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:49.419053 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:49.419062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:49.419570 (+ 508us) client_negotiation.cc:769] Sending connection context
0928 16:43:49.419906 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 16:43:49.420528 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:43:49.423695 31529 heartbeater.cc:347] Connected to a master server at 127.26.101.62:35147
I20260928 16:43:49.424000 31529 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:49.424676 31529 heartbeater.cc:511] Master 127.26.101.62:35147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:43:49.426249 31206 ts_manager.cc:194] Registered new tserver with Master: 1c061e5b828c483ba4d13df03389f50d (127.26.101.2:35769)
I20260928 16:43:49.427099 31206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:45723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:43:49.449808 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:49.794948 31534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:49.795369 31534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:49.795478 31534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:49.803535 31534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:49.803686 31534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:49.803817 31534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:49.814499 31534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:49.815948 31534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:49.817279 31534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:49.830786 31539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:49.830737 31542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:49.830803 31540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:49.831774 31534 server_base.cc:1034] running on GCE node
I20260928 16:43:49.832479 31534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:49.833662 31534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:49.834933 31534 hybrid_clock.cc:648] HybridClock initialized: now 1790613829834873 us; error 61 us; skew 500 ppm
I20260928 16:43:49.835364 31534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:49 dist-test-slave-t0tw krb5kdc[31161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:49.838867 31534 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:43:49.840394 31534 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:49.840559 31534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:49.840889 31534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:49.844132 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "3460dee4da5141b688a5a5be990082f2"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "149d2b435d97dd46591345afe14af2ef"
server_key_iv: "9dc3cf792fa8328ad77e9a326e0e28e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.845002 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "3460dee4da5141b688a5a5be990082f2"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "149d2b435d97dd46591345afe14af2ef"
server_key_iv: "9dc3cf792fa8328ad77e9a326e0e28e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.850536 31534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:43:49.854602 31548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.856472 31534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:43:49.856691 31534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "3460dee4da5141b688a5a5be990082f2"
format_stamp: "Formatted at 2026-09-28 16:43:49 on dist-test-slave-t0tw"
server_key: "149d2b435d97dd46591345afe14af2ef"
server_key_iv: "9dc3cf792fa8328ad77e9a326e0e28e6"
server_key_version: "encryptionkey@0"
I20260928 16:43:49.856927 31534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:49.870203 31534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:49.874953 31534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:49.875310 31534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:49.876329 31534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:49.877732 31534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:49.877830 31534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.877941 31534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:49.878012 31534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:49.909613 31534 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:44725
I20260928 16:43:49.909669 31662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:44725 every 8 connection(s)
I20260928 16:43:49.910480 31534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:49.919914 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31534
I20260928 16:43:49.920102 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:49.920547 27028 external_mini_cluster.cc:1428] Setting key 3eb7016977bdf76c73396f85cb60d8c5
Sep 28 16:43:49 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613829, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:49.924562 31395 heartbeater.cc:503] Master 127.26.101.62:35147 was elected leader, sending a full tablet report...
I20260928 16:43:49.930727 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.913064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36011 (local address 127.26.101.62:35147)
0928 16:43:49.913266 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.913270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.914515 (+ 1245us) server_negotiation.cc:407] Connection header received
0928 16:43:49.916418 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.916424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.916528 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.916619 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:49.918247 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.918993 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.920692 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.921210 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.921331 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.925486 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:49.925492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:49.925496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:49.925531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:49.927997 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.928692 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.928695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.928697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.928823 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.929167 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.929178 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.929180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.929447 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:49.929563 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.930225 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.930409 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:43:49.931792 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.913568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35147 (local address 127.26.101.3:36011)
0928 16:43:49.914336 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:49.914393 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:49.916127 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:49.916884 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:49.916893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:49.917440 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:49.918041 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.918061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.919171 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.919175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.920521 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.920534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.921454 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.921462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.921757 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.922668 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:49.922701 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:49.925243 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:49.928137 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:49.928144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:49.928159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:49.928593 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:49.928901 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:49.928904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:49.928906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:49.929081 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:49.929541 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:49.929547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:49.929903 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 16:43:49.930125 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:43:49.930542 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":291,"threads_started":1}
I20260928 16:43:49.933246 31663 heartbeater.cc:347] Connected to a master server at 127.26.101.62:35147
I20260928 16:43:49.933568 31663 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:49.934325 31663 heartbeater.cc:511] Master 127.26.101.62:35147 requested a full tablet report, sending...
I20260928 16:43:49.935946 31206 ts_manager.cc:194] Registered new tserver with Master: 3460dee4da5141b688a5a5be990082f2 (127.26.101.3:44725)
I20260928 16:43:49.936473 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:49.937016 31206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:36011
I20260928 16:43:49.954710 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.941783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.26.101.62:35147)
0928 16:43:49.941958 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.941962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.942509 (+ 547us) server_negotiation.cc:407] Connection header received
0928 16:43:49.942639 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.942643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.942708 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.942796 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:49.944231 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.944952 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.946278 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.946823 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.946924 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.949315 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:49.949320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:49.949324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:49.949365 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:49.952116 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.952823 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.952827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.952829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.952980 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:49.953290 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:49.953294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:49.953296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:49.953659 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:49.953854 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.954231 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.954429 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:43:49.959275 31206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:49.966140 31206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:49.992116 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.980141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.26.101.2:35769)
0928 16:43:49.981511 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.981522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.982305 (+ 783us) server_negotiation.cc:407] Connection header received
0928 16:43:49.982422 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.982430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.982720 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.982914 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:49.984438 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.986180 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.989382 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.990740 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.990878 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.991132 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.991574 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.991757 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":997,"threads_started":1}
I20260928 16:43:49.992868 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.979550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:35769 (local address 127.0.0.1:39208)
0928 16:43:49.981935 (+ 2385us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:49.981952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:49.982134 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:49.983700 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:49.983709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:49.983745 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:49.984191 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.984204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.986356 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.986363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.989193 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.989211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.991016 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.991023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.991261 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.991313 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 16:43:49.992108 (+ 795us) client_negotiation.cc:240] Negotiation successful
0928 16:43:49.992328 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2273,"spinlock_wait_cycles":10240,"thread_start_us":354,"threads_started":1}
I20260928 16:43:49.995466 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.979197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:42181 (local address 127.0.0.1:56304)
0928 16:43:49.981856 (+ 2659us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:49.981914 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:49.982133 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:49.987819 (+ 5686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:49.987824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:49.987848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:49.988355 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.988371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.990396 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.990402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.992655 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.992675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.994560 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.994566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.994738 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.994772 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:43:49.994870 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:43:49.995024 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"mutex_wait_us":4,"spinlock_wait_cycles":8192,"thread_start_us":2454,"threads_started":1}
I20260928 16:43:49.995769 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.979255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.26.101.1:42181)
0928 16:43:49.987253 (+ 7998us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.987261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.987291 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:43:49.987377 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.987384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.987597 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.988576 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:49.988807 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.990222 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.992847 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.994331 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.994483 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.994733 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.995265 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.995448 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7765,"thread_start_us":7615,"threads_started":1}
I20260928 16:43:49.996048 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.982430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.26.101.3:44725)
0928 16:43:49.983246 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:49.983256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:49.987182 (+ 3926us) server_negotiation.cc:407] Connection header received
0928 16:43:49.987314 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:49.987323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:49.987600 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:49.987798 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:49.988806 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.990064 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.993323 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.994775 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.994898 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.995147 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:49.995564 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 16:43:49.995723 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":85376,"thread_start_us":561,"threads_started":1}
I20260928 16:43:49.996299 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:49.982868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44725 (local address 127.0.0.1:59378)
0928 16:43:49.986857 (+ 3989us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:49.986880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:49.987016 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:49.988074 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:49.988078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:49.988105 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:49.988535 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.988547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.990250 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.990255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.993118 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:49.993134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:49.994984 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:49.994990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:49.995196 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:49.995227 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:43:49.995314 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:43:49.995444 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3837,"mutex_wait_us":480,"thread_start_us":3654,"threads_started":1}
I20260928 16:43:49.999560 31597 tablet_service.cc:1467] Processing CreateTablet for tablet 386c3dc69526414aa79a0de0ae33de92 (DEFAULT_TABLE table=test-table [id=319f66c0a47647f1aef8e633708d98e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:49.999541 31330 tablet_service.cc:1467] Processing CreateTablet for tablet 386c3dc69526414aa79a0de0ae33de92 (DEFAULT_TABLE table=test-table [id=319f66c0a47647f1aef8e633708d98e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:49.997161 31463 tablet_service.cc:1467] Processing CreateTablet for tablet 386c3dc69526414aa79a0de0ae33de92 (DEFAULT_TABLE table=test-table [id=319f66c0a47647f1aef8e633708d98e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:50.000799 31597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c3dc69526414aa79a0de0ae33de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:50.000885 31330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c3dc69526414aa79a0de0ae33de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:50.001385 31463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c3dc69526414aa79a0de0ae33de92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:50.012530 31681 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Bootstrap starting.
I20260928 16:43:50.016050 31683 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Bootstrap starting.
I20260928 16:43:50.016418 31682 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Bootstrap starting.
I20260928 16:43:50.016443 31681 tablet_bootstrap.cc:654] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:50.017786 31681 log.cc:824] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:50.020036 31683 tablet_bootstrap.cc:654] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:50.021215 31683 log.cc:824] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:50.021377 31681 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: No bootstrap required, opened a new log
I20260928 16:43:50.021617 31682 tablet_bootstrap.cc:654] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:50.021682 31681 ts_tablet_manager.cc:1397] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 16:43:50.022929 31682 log.cc:824] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:50.024164 31683 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: No bootstrap required, opened a new log
I20260928 16:43:50.024473 31683 ts_tablet_manager.cc:1397] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:43:50.025990 31682 tablet_bootstrap.cc:492] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: No bootstrap required, opened a new log
I20260928 16:43:50.026274 31682 ts_tablet_manager.cc:1397] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 16:43:50.032051 31683 raft_consensus.cc:348] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.032052 31681 raft_consensus.cc:348] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.032434 31683 raft_consensus.cc:374] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:50.032544 31683 raft_consensus.cc:729] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be97fd95353d4337838095bfbcfd0d91, State: Initialized, Role: FOLLOWER
I20260928 16:43:50.032536 31681 raft_consensus.cc:374] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:50.032639 31681 raft_consensus.cc:729] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3460dee4da5141b688a5a5be990082f2, State: Initialized, Role: FOLLOWER
I20260928 16:43:50.033162 31683 consensus_queue.cc:260] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.033365 31681 consensus_queue.cc:260] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.034709 31663 heartbeater.cc:503] Master 127.26.101.62:35147 was elected leader, sending a full tablet report...
I20260928 16:43:50.035291 31681 ts_tablet_manager.cc:1428] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260928 16:43:50.036631 31683 ts_tablet_manager.cc:1428] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260928 16:43:50.037252 31682 raft_consensus.cc:348] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.037706 31682 raft_consensus.cc:374] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:50.037814 31682 raft_consensus.cc:729] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c061e5b828c483ba4d13df03389f50d, State: Initialized, Role: FOLLOWER
I20260928 16:43:50.038405 31682 consensus_queue.cc:260] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.039731 31529 heartbeater.cc:503] Master 127.26.101.62:35147 was elected leader, sending a full tablet report...
I20260928 16:43:50.040470 31682 ts_tablet_manager.cc:1428] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 16:43:50.142010 31396 tablet.cc:2369] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:50.143834 31687 raft_consensus.cc:482] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:50.144102 31687 raft_consensus.cc:504] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.145864 31687 leader_election.cc:288] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3460dee4da5141b688a5a5be990082f2 (127.26.101.3:44725), 1c061e5b828c483ba4d13df03389f50d (127.26.101.2:35769)
W20260928 16:43:50.151744 31530 tablet.cc:2369] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:50.153172 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.146333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53995 (local address 127.26.101.3:44725)
0928 16:43:50.146593 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.146600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.147499 (+ 899us) server_negotiation.cc:407] Connection header received
0928 16:43:50.147697 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.147701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.147763 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.147841 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:50.148469 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.149223 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.151206 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.152123 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.152294 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.152375 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.152778 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.152912 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:43:50.153296 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.146404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:49005 (local address 127.26.101.2:35769)
0928 16:43:50.146617 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.146623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.147589 (+ 966us) server_negotiation.cc:407] Connection header received
0928 16:43:50.147771 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.147777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.147843 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.147915 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:43:50.148528 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.149304 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.151190 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.152375 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.152549 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.152635 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.152923 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.153070 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:43:50.153378 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.146373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:35769 (local address 127.26.101.1:49005)
0928 16:43:50.147466 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:50.147486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:50.147638 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:50.147948 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:50.147953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:50.147980 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:50.148375 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:50.148385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.149446 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.149451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:50.150995 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:50.151011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.152603 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.152609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:50.152769 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.152799 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:43:50.152879 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:43:50.152992 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":131,"spinlock_wait_cycles":1792,"thread_start_us":484,"threads_started":1}
I20260928 16:43:50.153774 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.146373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44725 (local address 127.26.101.1:53995)
0928 16:43:50.147367 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:50.147389 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:50.147541 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:50.147870 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:50.147875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:50.147904 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:43:50.148313 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:50.148327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.149356 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.149359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:50.151008 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:50.151021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.152348 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.152355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:50.152543 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.152582 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:43:50.152725 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:43:50.152882 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":39,"spinlock_wait_cycles":35584,"thread_start_us":574,"threads_started":1}
I20260928 16:43:50.154610 31483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c3dc69526414aa79a0de0ae33de92" candidate_uuid: "be97fd95353d4337838095bfbcfd0d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c061e5b828c483ba4d13df03389f50d" is_pre_election: true
I20260928 16:43:50.154812 31617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c3dc69526414aa79a0de0ae33de92" candidate_uuid: "be97fd95353d4337838095bfbcfd0d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3460dee4da5141b688a5a5be990082f2" is_pre_election: true
I20260928 16:43:50.155159 31483 raft_consensus.cc:2454] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be97fd95353d4337838095bfbcfd0d91 in term 0.
I20260928 16:43:50.155382 31617 raft_consensus.cc:2454] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be97fd95353d4337838095bfbcfd0d91 in term 0.
I20260928 16:43:50.155977 31282 leader_election.cc:302] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c061e5b828c483ba4d13df03389f50d, be97fd95353d4337838095bfbcfd0d91; no voters:
I20260928 16:43:50.156448 31687 raft_consensus.cc:2781] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:50.156603 31687 raft_consensus.cc:482] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:50.156698 31687 raft_consensus.cc:3037] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:50.158908 31687 raft_consensus.cc:504] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.159646 31687 leader_election.cc:288] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [CANDIDATE]: Term 1 election: Requested vote from peers 3460dee4da5141b688a5a5be990082f2 (127.26.101.3:44725), 1c061e5b828c483ba4d13df03389f50d (127.26.101.2:35769)
I20260928 16:43:50.160274 31617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c3dc69526414aa79a0de0ae33de92" candidate_uuid: "be97fd95353d4337838095bfbcfd0d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3460dee4da5141b688a5a5be990082f2"
I20260928 16:43:50.160544 31617 raft_consensus.cc:3037] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:50.161185 31483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c3dc69526414aa79a0de0ae33de92" candidate_uuid: "be97fd95353d4337838095bfbcfd0d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c061e5b828c483ba4d13df03389f50d"
I20260928 16:43:50.161414 31483 raft_consensus.cc:3037] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:43:50.162025 31664 tablet.cc:2369] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:50.164011 31617 raft_consensus.cc:2454] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be97fd95353d4337838095bfbcfd0d91 in term 1.
I20260928 16:43:50.164532 31483 raft_consensus.cc:2454] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be97fd95353d4337838095bfbcfd0d91 in term 1.
I20260928 16:43:50.164649 31285 leader_election.cc:302] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3460dee4da5141b688a5a5be990082f2, be97fd95353d4337838095bfbcfd0d91; no voters:
I20260928 16:43:50.165057 31687 raft_consensus.cc:2781] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:50.165549 31687 raft_consensus.cc:686] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 1 LEADER]: Becoming Leader. State: Replica: be97fd95353d4337838095bfbcfd0d91, State: Running, Role: LEADER
I20260928 16:43:50.166085 31687 consensus_queue.cc:237] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } }
I20260928 16:43:50.171044 31206 catalog_manager.cc:5495] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to be97fd95353d4337838095bfbcfd0d91 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "be97fd95353d4337838095bfbcfd0d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be97fd95353d4337838095bfbcfd0d91" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:50.230789 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.225467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.26.101.1:42181)
0928 16:43:50.225694 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.225700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.225782 (+ 82us) server_negotiation.cc:407] Connection header received
0928 16:43:50.225965 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.225968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.226072 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.226205 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:43:50.226587 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.227371 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.228722 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.229156 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.229266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.229686 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:43:50.230221 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:43:50.230343 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.230448 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.230521 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:43:50.239328 31483 raft_consensus.cc:1261] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 1 FOLLOWER]: Refusing update from remote peer be97fd95353d4337838095bfbcfd0d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:50.239375 31617 raft_consensus.cc:1261] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 1 FOLLOWER]: Refusing update from remote peer be97fd95353d4337838095bfbcfd0d91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:50.240356 31692 consensus_queue.cc:1035] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3460dee4da5141b688a5a5be990082f2" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:50.240797 31687 consensus_queue.cc:1035] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c061e5b828c483ba4d13df03389f50d" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 35769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:50.251933 31695 mvcc.cc:205] Tried to move back new op lower bound from 7334354248649961472 to 7334354248364412928. Current Snapshot: MvccSnapshot[applied={T|T < 7334354248649961472}]
I20260928 16:43:50.262092 31696 mvcc.cc:205] Tried to move back new op lower bound from 7334354248649961472 to 7334354248364412928. Current Snapshot: MvccSnapshot[applied={T|T < 7334354248649961472 or (T in {7334354248649961472})}]
I20260928 16:43:50.262498 31697 mvcc.cc:205] Tried to move back new op lower bound from 7334354248649961472 to 7334354248364412928. Current Snapshot: MvccSnapshot[applied={T|T < 7334354248649961472 or (T in {7334354248649961472})}]
I20260928 16:43:50.268656 31206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:50.268934 31206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:50.273236 31206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da7be27a1d13437a98c603533cf8b6ee: Sending DeleteTablet for 3 replicas of tablet 386c3dc69526414aa79a0de0ae33de92
I20260928 16:43:50.274628 31597 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c3dc69526414aa79a0de0ae33de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:59378
I20260928 16:43:50.274717 31463 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c3dc69526414aa79a0de0ae33de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:39208
I20260928 16:43:50.274847 31330 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c3dc69526414aa79a0de0ae33de92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:56304
I20260928 16:43:50.277057 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31267
I20260928 16:43:50.305406 31706 tablet_replica.cc:330] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: stopping tablet replica
I20260928 16:43:50.306380 31705 tablet_replica.cc:330] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: stopping tablet replica
I20260928 16:43:50.306869 31705 raft_consensus.cc:2229] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:50.307478 31705 raft_consensus.cc:2258] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:50.308449 31707 tablet_replica.cc:330] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: stopping tablet replica
I20260928 16:43:50.308952 31707 raft_consensus.cc:2229] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:43:50.309756 31707 raft_consensus.cc:2258] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:50.310253 31705 ts_tablet_manager.cc:1905] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:50.310794 31706 raft_consensus.cc:2229] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:50.311439 31706 raft_consensus.cc:2258] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:50.312965 31707 ts_tablet_manager.cc:1905] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:50.314524 31706 ts_tablet_manager.cc:1905] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:43:50 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:43:50.318542 31705 ts_tablet_manager.cc:1918] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:50.318733 31705 log.cc:1196] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/386c3dc69526414aa79a0de0ae33de92
I20260928 16:43:50.319316 31705 ts_tablet_manager.cc:1939] T 386c3dc69526414aa79a0de0ae33de92 P 1c061e5b828c483ba4d13df03389f50d: Deleting consensus metadata
I20260928 16:43:50.320375 31707 ts_tablet_manager.cc:1918] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:50.320592 31707 log.cc:1196] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/386c3dc69526414aa79a0de0ae33de92
I20260928 16:43:50.321218 31707 ts_tablet_manager.cc:1939] T 386c3dc69526414aa79a0de0ae33de92 P be97fd95353d4337838095bfbcfd0d91: Deleting consensus metadata
I20260928 16:43:50.321514 31190 catalog_manager.cc:4841] TS 1c061e5b828c483ba4d13df03389f50d (127.26.101.2:35769): tablet 386c3dc69526414aa79a0de0ae33de92 (table test-table [id=319f66c0a47647f1aef8e633708d98e0]) successfully deleted
I20260928 16:43:50.323235 31193 catalog_manager.cc:4841] TS be97fd95353d4337838095bfbcfd0d91 (127.26.101.1:42181): tablet 386c3dc69526414aa79a0de0ae33de92 (table test-table [id=319f66c0a47647f1aef8e633708d98e0]) successfully deleted
I20260928 16:43:50.325771 31706 ts_tablet_manager.cc:1918] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:50.325937 31706 log.cc:1196] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/386c3dc69526414aa79a0de0ae33de92
I20260928 16:43:50.326359 31706 ts_tablet_manager.cc:1939] T 386c3dc69526414aa79a0de0ae33de92 P 3460dee4da5141b688a5a5be990082f2: Deleting consensus metadata
I20260928 16:43:50.328141 31193 catalog_manager.cc:4841] TS 3460dee4da5141b688a5a5be990082f2 (127.26.101.3:44725): tablet 386c3dc69526414aa79a0de0ae33de92 (table test-table [id=319f66c0a47647f1aef8e633708d98e0]) successfully deleted
I20260928 16:43:50.329190 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.286511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.26.101.1:42181)
0928 16:43:50.286756 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.286764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.302160 (+ 15396us) server_negotiation.cc:407] Connection header received
0928 16:43:50.302683 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.302689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.302785 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.302855 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:50.305729 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.306957 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.309686 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.310391 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.310523 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.321504 (+ 10981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:50.321523 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:50.321542 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:50.321597 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:50.325798 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.326842 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.326852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.326855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.327045 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.327557 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.327562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.327565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.328095 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:50.328459 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.328691 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.328881 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:43:50.330001 31390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:50.431905 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31267
I20260928 16:43:50.449067 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31401
Sep 28 16:43:50 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:43:50.463960 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.449794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.26.101.2:35769)
0928 16:43:50.449961 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.449965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.450007 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:43:50.450155 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.450159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.450232 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.450317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:50.451711 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.452466 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.453695 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.454079 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.454175 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.458192 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:50.458210 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:50.458226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:50.458264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:50.461190 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.461928 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.461936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.461939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.462087 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.462372 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.462375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.462377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.462755 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:50.462984 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.463495 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.463706 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:43:50.464623 31524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:50.553124 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31401
I20260928 16:43:50.568511 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31534
Sep 28 16:43:50 dist-test-slave-t0tw krb5kdc[31161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:43:50.583642 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:50.569227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.26.101.3:44725)
0928 16:43:50.569404 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:50.569409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:50.569428 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:43:50.569564 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:50.569569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:50.569679 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:50.569767 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:50.571045 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.571822 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.573042 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:50.573440 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:50.573537 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:50.577765 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:50.577783 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:50.577801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:50.577838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:50.580864 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.581527 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.581535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.581538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.581674 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:50.582054 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:50.582058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:50.582060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:50.582457 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:50.582750 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:50.583069 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:43:50.583332 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:43:50.584263 31658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:50.676529 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31534
I20260928 16:43:50.691762 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31174
I20260928 16:43:50.692924 31236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:50.801963 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31174
2026-09-28T16:43:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4192 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T16:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:50Z Disabled control of system clock
I20260928 16:43:50.865875 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:33377
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:43:51.193171 31716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:51.193483 31716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:51.193601 31716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:51.200949 31716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:51.201085 31716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:51.201126 31716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:51.201205 31716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:51.201265 31716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:51.210676 31716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:33377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:51.212092 31716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:51.213992 31716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:51.225662 31721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.225617 31724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.225629 31722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:51.226663 31716 server_base.cc:1034] running on GCE node
I20260928 16:43:51.227505 31716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:51.228916 31716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:51.230149 31716 hybrid_clock.cc:648] HybridClock initialized: now 1790613831230094 us; error 42 us; skew 500 ppm
I20260928 16:43:51.230508 31716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:51.232852 31716 webserver.cc:460] Webserver started at http://127.26.101.62:37299/ using document root <none> and password file <none>
I20260928 16:43:51.233634 31716 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:51.233733 31716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:51.234004 31716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:51.236939 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "e050520376434c03b52252a94dfa8caa"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "3b881587121ca89aea48821b43c62b88"
server_key_iv: "b2b068d1d49c23c53eae698f68e64919"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.237681 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "e050520376434c03b52252a94dfa8caa"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "3b881587121ca89aea48821b43c62b88"
server_key_iv: "b2b068d1d49c23c53eae698f68e64919"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.242784 31716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:43:51.246380 31730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:51.247918 31716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:43:51.248093 31716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "e050520376434c03b52252a94dfa8caa"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "3b881587121ca89aea48821b43c62b88"
server_key_iv: "b2b068d1d49c23c53eae698f68e64919"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.248317 31716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:51.254251 31716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:51.255051 31716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:51.255398 31716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:51.272388 31716 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:33377
I20260928 16:43:51.272423 31781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:33377 every 8 connection(s)
I20260928 16:43:51.273772 31716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:51.274557 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31716
I20260928 16:43:51.274740 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:51.275588 27028 external_mini_cluster.cc:1428] Setting key 11a23fad383682b0c062a83169ec01a2
I20260928 16:43:51.278340 31782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:51.284261 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:51.277629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.26.101.62:33377)
0928 16:43:51.278779 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:51.278791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:51.278834 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:43:51.280550 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:51.280574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:51.281013 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:51.281410 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:51.282482 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:51.282488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:51.282504 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:51.282520 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:51.282621 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:51.282638 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:51.282868 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:51.283034 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 16:43:51.283317 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":371,"threads_started":1}
I20260928 16:43:51.289861 31782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: Bootstrap starting.
I20260928 16:43:51.293205 31782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:51.294250 31782 log.cc:824] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:51.297276 31782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: No bootstrap required, opened a new log
I20260928 16:43:51.305088 31782 raft_consensus.cc:348] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } }
I20260928 16:43:51.305492 31782 raft_consensus.cc:374] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:51.305608 31782 raft_consensus.cc:729] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e050520376434c03b52252a94dfa8caa, State: Initialized, Role: FOLLOWER
I20260928 16:43:51.306247 31782 consensus_queue.cc:260] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } }
I20260928 16:43:51.306479 31782 raft_consensus.cc:388] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:51.306617 31782 raft_consensus.cc:482] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:51.306784 31782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:51.309054 31782 raft_consensus.cc:504] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } }
I20260928 16:43:51.309588 31782 leader_election.cc:302] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e050520376434c03b52252a94dfa8caa; no voters:
I20260928 16:43:51.310294 31782 leader_election.cc:288] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:51.310417 31787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:51.311571 31787 raft_consensus.cc:686] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [term 1 LEADER]: Becoming Leader. State: Replica: e050520376434c03b52252a94dfa8caa, State: Running, Role: LEADER
I20260928 16:43:51.312120 31787 consensus_queue.cc:237] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } }
I20260928 16:43:51.312541 31782 sys_catalog.cc:564] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:51.314690 31789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e050520376434c03b52252a94dfa8caa. Latest consensus state: current_term: 1 leader_uuid: "e050520376434c03b52252a94dfa8caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } } }
I20260928 16:43:51.314697 31788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e050520376434c03b52252a94dfa8caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e050520376434c03b52252a94dfa8caa" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 33377 } } }
I20260928 16:43:51.315109 31789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:51.315111 31788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:51.316033 31793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:51.321487 31793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:51.332506 31793 catalog_manager.cc:1347] Generated new cluster ID: 245d2c8a695b4435bee51f91d931c40a
I20260928 16:43:51.332640 31793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:51.368924 31793 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:51.369347 31793 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:51.380017 31793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: Generated new TSK 0
I20260928 16:43:51.381202 31793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:51.400022 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--builtin_ntp_servers=127.26.101.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:43:51.724215 31806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:51.724525 31806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:51.724639 31806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:51.731971 31806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:51.732084 31806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:51.732221 31806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:51.741683 31806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:51.743037 31806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:51.744864 31806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:51.757889 31813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.757918 31812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:51.757846 31815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:51.758805 31806 server_base.cc:1034] running on GCE node
I20260928 16:43:51.759450 31806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:51.760527 31806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:51.761750 31806 hybrid_clock.cc:648] HybridClock initialized: now 1790613831761678 us; error 65 us; skew 500 ppm
I20260928 16:43:51.762082 31806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:51.764535 31806 webserver.cc:460] Webserver started at http://127.26.101.1:35027/ using document root <none> and password file <none>
I20260928 16:43:51.765246 31806 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:51.765352 31806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:51.765690 31806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:51.768702 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "f2c1df160130ad77e6582677eb1cc3f0"
server_key_iv: "0a1d2365430344139933ff63e6a316d7"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.769452 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "f2c1df160130ad77e6582677eb1cc3f0"
server_key_iv: "0a1d2365430344139933ff63e6a316d7"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.774482 31806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:43:51.778017 31822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:51.779385 31806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:51.779595 31806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe"
format_stamp: "Formatted at 2026-09-28 16:43:51 on dist-test-slave-t0tw"
server_key: "f2c1df160130ad77e6582677eb1cc3f0"
server_key_iv: "0a1d2365430344139933ff63e6a316d7"
server_key_version: "encryptionkey@0"
I20260928 16:43:51.779800 31806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:51.785693 31806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:51.786550 31806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:51.786808 31806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:51.787842 31806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:51.789343 31806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:51.789439 31806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:51.789553 31806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:51.789623 31806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:51.820150 31806 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:44989
I20260928 16:43:51.820219 31935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:44989 every 8 connection(s)
I20260928 16:43:51.821496 31806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:51.828897 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31806
I20260928 16:43:51.829078 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:51.829695 27028 external_mini_cluster.cc:1428] Setting key d8ebf53c2b1a875dcc720c5dc136e9da
I20260928 16:43:51.829754 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:51.824011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53353 (local address 127.26.101.62:33377)
0928 16:43:51.824429 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:51.824433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:51.825744 (+ 1311us) server_negotiation.cc:407] Connection header received
0928 16:43:51.827474 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:51.827480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:51.827552 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:51.827651 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:51.828955 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:51.828960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:51.828964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:51.828984 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:51.829055 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:51.829071 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:51.829195 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:51.829385 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:43:51.829455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":215,"threads_started":1}
I20260928 16:43:51.830664 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:51.824609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:33377 (local address 127.26.101.1:53353)
0928 16:43:51.825583 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:51.825622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:51.827196 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:51.827902 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:51.827910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:51.828014 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:51.828652 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:51.828654 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:51.828693 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:51.828780 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:51.829187 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:51.829197 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:51.829205 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:43:51.829455 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 16:43:51.829717 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":302,"threads_started":1}
I20260928 16:43:51.832219 31936 heartbeater.cc:347] Connected to a master server at 127.26.101.62:33377
I20260928 16:43:51.832556 31936 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:51.832587 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--builtin_ntp_servers=127.26.101.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:43:51.833261 31936 heartbeater.cc:511] Master 127.26.101.62:33377 requested a full tablet report, sending...
I20260928 16:43:51.835551 31746 ts_manager.cc:194] Registered new tserver with Master: 3e520a3d28fe40a8ba55bd2f49a76dfe (127.26.101.1:44989)
W20260928 16:43:52.158674 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:52.158998 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:52.159161 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:52.168622 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:52.168761 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:52.168926 31941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:43:52.178418 31941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:52.179862 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:52.181802 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:52.196091 31947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:52.196126 31946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:52.196148 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:52.196967 31941 server_base.cc:1034] running on GCE node
I20260928 16:43:52.197610 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:52.198720 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:52.199921 31941 hybrid_clock.cc:648] HybridClock initialized: now 1790613832199867 us; error 48 us; skew 500 ppm
I20260928 16:43:52.200248 31941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:52.202801 31941 webserver.cc:460] Webserver started at http://127.26.101.2:35291/ using document root <none> and password file <none>
I20260928 16:43:52.203578 31941 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:52.203703 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:52.204053 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:52.207055 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "a744a6e29eb9466486edc57afd937343"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "1b1cac5bbb08a59054ce6e4241a44b52"
server_key_iv: "c9d4322f95e4db0f35a27fc595f3427f"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.207911 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "a744a6e29eb9466486edc57afd937343"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "1b1cac5bbb08a59054ce6e4241a44b52"
server_key_iv: "c9d4322f95e4db0f35a27fc595f3427f"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.213466 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:52.217312 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.218770 31941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:52.218943 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "a744a6e29eb9466486edc57afd937343"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "1b1cac5bbb08a59054ce6e4241a44b52"
server_key_iv: "c9d4322f95e4db0f35a27fc595f3427f"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.219206 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:52.225560 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:52.226496 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:52.226790 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:52.227963 31941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:52.229638 31941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:52.229754 31941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.229875 31941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:52.229946 31941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.261979 31941 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:40651
I20260928 16:43:52.262086 32068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:40651 every 8 connection(s)
I20260928 16:43:52.263478 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:43:52.272688 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 31941
I20260928 16:43:52.272886 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:43:52.272917 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.266444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:48743 (local address 127.26.101.62:33377)
0928 16:43:52.266686 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.266691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.268395 (+ 1704us) server_negotiation.cc:407] Connection header received
0928 16:43:52.270216 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.270223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.270302 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.270402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.272002 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.272007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.272011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.272033 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.272072 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.272085 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.272200 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.272470 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.272551 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:43:52.273497 27028 external_mini_cluster.cc:1428] Setting key 3136867191228fba7ee444686b8e6178
I20260928 16:43:52.274178 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.267051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:33377 (local address 127.26.101.2:48743)
0928 16:43:52.268216 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:52.268258 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:52.269883 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:52.270691 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:52.270701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:52.270812 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:52.271552 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:52.271554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:52.271622 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:52.271757 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:52.272213 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:52.272225 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:52.272233 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:43:52.272550 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 16:43:52.272876 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":333,"threads_started":1}
I20260928 16:43:52.275880 32069 heartbeater.cc:347] Connected to a master server at 127.26.101.62:33377
I20260928 16:43:52.276228 32069 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:52.276803 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--builtin_ntp_servers=127.26.101.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:43:52.276880 32069 heartbeater.cc:511] Master 127.26.101.62:33377 requested a full tablet report, sending...
I20260928 16:43:52.278703 31746 ts_manager.cc:194] Registered new tserver with Master: a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651)
W20260928 16:43:52.605983 32072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:52.606297 32072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:52.606426 32072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:52.613773 32072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:52.613894 32072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:52.614053 32072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:43:52.623252 32072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:52.624593 32072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:52.626292 32072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:52.639062 32077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:52.639017 32078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:52.638988 32080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:52.639612 32072 server_base.cc:1034] running on GCE node
I20260928 16:43:52.640265 32072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:52.641280 32072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:52.642493 32072 hybrid_clock.cc:648] HybridClock initialized: now 1790613832642445 us; error 48 us; skew 500 ppm
I20260928 16:43:52.642810 32072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:52.645088 32072 webserver.cc:460] Webserver started at http://127.26.101.3:34861/ using document root <none> and password file <none>
I20260928 16:43:52.645773 32072 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:52.645881 32072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:52.646209 32072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:52.649127 32072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "48fb050af13148fcbbbeffde7cfbf4c0"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "7b2961fe43909966507f75a89574e788"
server_key_iv: "d0f52f907e00fbcdf728da1a9a6beaae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.649894 32072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "48fb050af13148fcbbbeffde7cfbf4c0"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "7b2961fe43909966507f75a89574e788"
server_key_iv: "d0f52f907e00fbcdf728da1a9a6beaae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.655167 32072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:43:52.658705 32086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.660036 32072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:43:52.660216 32072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "48fb050af13148fcbbbeffde7cfbf4c0"
format_stamp: "Formatted at 2026-09-28 16:43:52 on dist-test-slave-t0tw"
server_key: "7b2961fe43909966507f75a89574e788"
server_key_iv: "d0f52f907e00fbcdf728da1a9a6beaae"
server_key_version: "encryptionkey@0"
I20260928 16:43:52.660434 32072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:52.666028 32072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:52.666826 32072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:52.667156 32072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:52.668051 32072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:52.669461 32072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:52.669556 32072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.669670 32072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:52.669739 32072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:52.700119 32072 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:40009
I20260928 16:43:52.700191 32199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:40009 every 8 connection(s)
I20260928 16:43:52.701455 32072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:43:52.708737 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32072
I20260928 16:43:52.708907 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:43:52.709450 27028 external_mini_cluster.cc:1428] Setting key 51034bd469bab34c7a555f82bf5ecda2
I20260928 16:43:52.709584 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.704182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:43361 (local address 127.26.101.62:33377)
0928 16:43:52.704374 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.704378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.705739 (+ 1361us) server_negotiation.cc:407] Connection header received
0928 16:43:52.707440 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.707444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.707500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.707588 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.708868 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.708874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.708878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.708900 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.708945 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.708961 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.709085 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.709230 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.709294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:43:52.710405 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.704703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:33377 (local address 127.26.101.3:43361)
0928 16:43:52.705597 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:52.705634 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:52.707204 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:52.707804 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:52.707812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:52.707906 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:52.708524 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:52.708527 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:52.708593 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:52.708682 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:52.709049 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:52.709057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:52.709064 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:43:52.709275 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 16:43:52.709473 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260928 16:43:52.711790 32200 heartbeater.cc:347] Connected to a master server at 127.26.101.62:33377
I20260928 16:43:52.712072 32200 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:52.712704 32200 heartbeater.cc:511] Master 127.26.101.62:33377 requested a full tablet report, sending...
I20260928 16:43:52.714021 31746 ts_manager.cc:194] Registered new tserver with Master: 48fb050af13148fcbbbeffde7cfbf4c0 (127.26.101.3:40009)
I20260928 16:43:52.724834 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:43:52.731496 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.729195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.26.101.62:33377)
0928 16:43:52.729348 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.729353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.729865 (+ 512us) server_negotiation.cc:407] Connection header received
0928 16:43:52.730002 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.730005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.730067 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.730149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.731005 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.731008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.731012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.731023 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.731048 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.731058 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.731195 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.731248 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.731294 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:43:52.735258 31746 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:43:52.742141 31746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:43:52.770406 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.754250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40651 (local address 127.0.0.1:47672)
0928 16:43:52.755175 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:52.755220 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:52.755370 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:52.767247 (+ 11877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:52.767257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:52.767386 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:52.768414 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:52.768418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:52.768461 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:52.768553 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:52.769553 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:52.769580 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:52.769589 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:43:52.769772 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:43:52.769878 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":422,"threads_started":1}
I20260928 16:43:52.770987 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.754609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.26.101.2:40651)
0928 16:43:52.766481 (+ 11872us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.766493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.766533 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:43:52.766632 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.766642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.766952 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.768010 (+ 1058us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.769288 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.769297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.769315 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.769338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.769393 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.769413 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.770403 (+ 990us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.770483 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.770592 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11664,"thread_start_us":11524,"threads_started":1}
I20260928 16:43:52.772698 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.755049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59838)
0928 16:43:52.768102 (+ 13053us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:52.768121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:52.768226 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:52.769867 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:52.769871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:52.769894 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:52.770677 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:52.770681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:52.770700 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:52.770746 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:52.771785 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:52.771790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:52.771796 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:43:52.772180 (+ 384us) client_negotiation.cc:240] Negotiation successful
0928 16:43:52.772234 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12927,"thread_start_us":12547,"threads_started":1}
I20260928 16:43:52.772907 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.755556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.26.101.1:44989)
0928 16:43:52.768721 (+ 13165us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.768730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.768761 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:43:52.768829 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.768836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.769066 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.769219 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.770928 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.770937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.770951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.770978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.771036 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.771058 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.772383 (+ 1325us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.772472 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.772591 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12983,"thread_start_us":14509,"threads_started":1}
I20260928 16:43:52.773164 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.754750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40009 (local address 127.0.0.1:54436)
0928 16:43:52.765508 (+ 10758us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:52.765538 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:52.765705 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:52.771396 (+ 5691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:52.771400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:52.771423 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:52.772012 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:52.772015 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:52.772032 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:52.772074 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:52.772527 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:52.772533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:52.772542 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:43:52.772671 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:43:52.772725 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10586,"thread_start_us":11047,"threads_started":1}
I20260928 16:43:52.773952 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:52.755355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.26.101.3:40009)
0928 16:43:52.770606 (+ 15251us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:52.770614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:52.770646 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:43:52.770708 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:52.770715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:52.770943 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:52.771354 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:52.772213 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:52.772220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:52.772233 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:52.772250 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:52.772299 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:52.772398 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:52.773419 (+ 1021us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:52.773485 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 16:43:52.773587 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15069,"thread_start_us":15730,"threads_started":1}
I20260928 16:43:52.775770 32003 tablet_service.cc:1467] Processing CreateTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 (DEFAULT_TABLE table=test-table [id=26bb42f9c44143378bea473390120695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:52.777364 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df31bda66a4a6fb7cfb5d500dccba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:52.777328 31871 tablet_service.cc:1467] Processing CreateTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 (DEFAULT_TABLE table=test-table [id=26bb42f9c44143378bea473390120695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:52.777405 32134 tablet_service.cc:1467] Processing CreateTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 (DEFAULT_TABLE table=test-table [id=26bb42f9c44143378bea473390120695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:43:52.778488 31871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df31bda66a4a6fb7cfb5d500dccba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:52.778767 32134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df31bda66a4a6fb7cfb5d500dccba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:52.789903 32218 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Bootstrap starting.
I20260928 16:43:52.793479 32218 tablet_bootstrap.cc:654] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:52.794485 32218 log.cc:824] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:52.795462 32220 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Bootstrap starting.
I20260928 16:43:52.797266 32219 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Bootstrap starting.
I20260928 16:43:52.799104 32218 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: No bootstrap required, opened a new log
I20260928 16:43:52.799270 32220 tablet_bootstrap.cc:654] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:52.799428 32218 ts_tablet_manager.cc:1397] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260928 16:43:52.800803 32220 log.cc:824] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:52.800906 32219 tablet_bootstrap.cc:654] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:52.801952 32219 log.cc:824] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:52.803848 32220 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: No bootstrap required, opened a new log
I20260928 16:43:52.804180 32220 ts_tablet_manager.cc:1397] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 16:43:52.804912 32219 tablet_bootstrap.cc:492] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: No bootstrap required, opened a new log
I20260928 16:43:52.805245 32219 ts_tablet_manager.cc:1397] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:43:52.808750 32218 raft_consensus.cc:348] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.809270 32218 raft_consensus.cc:374] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:52.809386 32218 raft_consensus.cc:729] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e520a3d28fe40a8ba55bd2f49a76dfe, State: Initialized, Role: FOLLOWER
I20260928 16:43:52.810045 32218 consensus_queue.cc:260] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.813983 31936 heartbeater.cc:503] Master 127.26.101.62:33377 was elected leader, sending a full tablet report...
I20260928 16:43:52.814771 32218 ts_tablet_manager.cc:1428] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260928 16:43:52.814589 32220 raft_consensus.cc:348] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.815034 32220 raft_consensus.cc:374] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:52.815178 32220 raft_consensus.cc:729] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fb050af13148fcbbbeffde7cfbf4c0, State: Initialized, Role: FOLLOWER
I20260928 16:43:52.815836 32220 consensus_queue.cc:260] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.817639 32219 raft_consensus.cc:348] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.818373 32219 raft_consensus.cc:374] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:52.818538 32219 raft_consensus.cc:729] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a744a6e29eb9466486edc57afd937343, State: Initialized, Role: FOLLOWER
I20260928 16:43:52.819797 32219 consensus_queue.cc:260] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:52.822165 32200 heartbeater.cc:503] Master 127.26.101.62:33377 was elected leader, sending a full tablet report...
I20260928 16:43:52.822857 32220 ts_tablet_manager.cc:1428] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Time spent starting tablet: real 0.019s user 0.015s sys 0.002s
W20260928 16:43:52.823669 31937 tablet.cc:2369] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:52.826083 32069 heartbeater.cc:503] Master 127.26.101.62:33377 was elected leader, sending a full tablet report...
I20260928 16:43:52.826079 32219 ts_tablet_manager.cc:1428] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Time spent starting tablet: real 0.021s user 0.008s sys 0.008s
W20260928 16:43:52.953073 32201 tablet.cc:2369] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:43:53.015569 32070 tablet.cc:2369] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:43:53.178702 32224 raft_consensus.cc:482] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:53.179059 32224 raft_consensus.cc:504] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:53.180852 32224 leader_election.cc:288] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651), 48fb050af13148fcbbbeffde7cfbf4c0 (127.26.101.3:40009)
I20260928 16:43:53.182861 32225 raft_consensus.cc:482] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:43:53.183259 32225 raft_consensus.cc:504] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:53.184525 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.181368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:43707 (local address 127.26.101.3:40009)
0928 16:43:53.181566 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.181570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.182393 (+ 823us) server_negotiation.cc:407] Connection header received
0928 16:43:53.182563 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.182569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.182621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.182703 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.183926 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.183932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.183936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.183954 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.183985 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.183997 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.184144 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.184198 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.184263 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:43:53.184725 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.181321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40009 (local address 127.26.101.1:43707)
0928 16:43:53.182255 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.182271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:53.182428 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:53.182754 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:53.182759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:53.182783 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:53.183721 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:53.183725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:53.183744 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:53.183791 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:53.184108 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:53.184112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:53.184117 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:53.184242 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 16:43:53.184293 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":543,"threads_started":1}
I20260928 16:43:53.185194 32225 leader_election.cc:288] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651), 3e520a3d28fe40a8ba55bd2f49a76dfe (127.26.101.1:44989)
I20260928 16:43:53.185773 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.181299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:60433 (local address 127.26.101.2:40651)
0928 16:43:53.181481 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.181487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.182765 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 16:43:53.183029 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.183035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.183143 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.183234 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.185031 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.185036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.185040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.185062 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.185102 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.185118 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.185233 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.185425 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.185496 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:43:53.186595 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.181760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40651 (local address 127.26.101.1:60433)
0928 16:43:53.182643 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.182661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:53.182872 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:53.183398 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:53.183403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:53.183426 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:53.184792 (+ 1366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:53.184796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:53.184814 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:53.184857 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:53.185288 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:53.185292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:53.185297 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:53.186115 (+ 818us) client_negotiation.cc:240] Negotiation successful
0928 16:43:53.186169 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":536,"threads_started":1}
I20260928 16:43:53.186587 32154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" is_pre_election: true
I20260928 16:43:53.187611 32024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a744a6e29eb9466486edc57afd937343" is_pre_election: true
I20260928 16:43:53.188284 32024 raft_consensus.cc:2454] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in term 0.
I20260928 16:43:53.188759 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.185673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:55273 (local address 127.26.101.2:40651)
0928 16:43:53.186010 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.186015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.186408 (+ 393us) server_negotiation.cc:407] Connection header received
0928 16:43:53.186465 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.186469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.186548 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.186640 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.187772 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.187782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.187787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.187808 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.187851 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.187867 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.188009 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.188317 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.188379 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"threads_started":1}
I20260928 16:43:53.187093 32154 raft_consensus.cc:2454] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in term 0.
I20260928 16:43:53.189509 31826 leader_election.cc:302] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e520a3d28fe40a8ba55bd2f49a76dfe, a744a6e29eb9466486edc57afd937343; no voters:
I20260928 16:43:53.189949 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.185731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40651 (local address 127.26.101.3:55273)
0928 16:43:53.186193 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.186209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:53.186309 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:53.186804 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:53.186809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:53.186838 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:53.187544 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:53.187547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:53.187565 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:53.187614 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:53.188069 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:53.188075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:53.188081 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:43:53.188181 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:43:53.188238 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260928 16:43:53.189941 32224 raft_consensus.cc:2781] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:53.190124 32224 raft_consensus.cc:482] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:53.190228 32224 raft_consensus.cc:3037] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:53.192232 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.185896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.26.101.3:55105)
0928 16:43:53.190168 (+ 4272us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.190188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:53.190286 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:53.190683 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:53.190689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:53.190713 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:53.191312 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:43:53.191315 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:53.191333 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:43:53.191378 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:53.191797 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:53.191802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:53.191807 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:43:53.191898 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:43:53.191950 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4158,"threads_started":1}
I20260928 16:43:53.192335 32024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a744a6e29eb9466486edc57afd937343" is_pre_election: true
I20260928 16:43:53.192544 32024 raft_consensus.cc:2454] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fb050af13148fcbbbeffde7cfbf4c0 in term 0.
I20260928 16:43:53.192935 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.185488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:55105 (local address 127.26.101.1:44989)
0928 16:43:53.186064 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.186069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.190390 (+ 4321us) server_negotiation.cc:407] Connection header received
0928 16:43:53.190460 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.190465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.190533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.190627 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.191547 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.191553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.191558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.191582 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.191617 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.191632 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.191744 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.192580 (+ 836us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.192647 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260928 16:43:53.193048 32090 leader_election.cc:302] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fb050af13148fcbbbeffde7cfbf4c0, a744a6e29eb9466486edc57afd937343; no voters:
I20260928 16:43:53.193492 32225 raft_consensus.cc:2781] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:43:53.193666 32225 raft_consensus.cc:482] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:43:53.193760 32225 raft_consensus.cc:3037] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:53.194324 32224 raft_consensus.cc:504] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:53.195034 32224 leader_election.cc:288] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [CANDIDATE]: Term 1 election: Requested vote from peers a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651), 48fb050af13148fcbbbeffde7cfbf4c0 (127.26.101.3:40009)
I20260928 16:43:53.195331 32024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a744a6e29eb9466486edc57afd937343"
I20260928 16:43:53.195555 32024 raft_consensus.cc:3037] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:53.195890 32154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48fb050af13148fcbbbeffde7cfbf4c0"
I20260928 16:43:53.196419 31891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" is_pre_election: true
I20260928 16:43:53.196882 31891 raft_consensus.cc:2379] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48fb050af13148fcbbbeffde7cfbf4c0 in current term 1: Already voted for candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in this term.
I20260928 16:43:53.196877 32225 raft_consensus.cc:504] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:53.197223 32154 raft_consensus.cc:2379] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in current term 1: Already voted for candidate 48fb050af13148fcbbbeffde7cfbf4c0 in this term.
I20260928 16:43:53.197760 32225 leader_election.cc:288] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [CANDIDATE]: Term 1 election: Requested vote from peers a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651), 3e520a3d28fe40a8ba55bd2f49a76dfe (127.26.101.1:44989)
I20260928 16:43:53.198555 31891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe"
I20260928 16:43:53.198789 31891 raft_consensus.cc:2379] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 48fb050af13148fcbbbeffde7cfbf4c0 in current term 1: Already voted for candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in this term.
I20260928 16:43:53.198720 32023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50df31bda66a4a6fb7cfb5d500dccba4" candidate_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a744a6e29eb9466486edc57afd937343"
I20260928 16:43:53.199540 32024 raft_consensus.cc:2454] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e520a3d28fe40a8ba55bd2f49a76dfe in term 1.
I20260928 16:43:53.199584 32090 leader_election.cc:302] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48fb050af13148fcbbbeffde7cfbf4c0; no voters: 3e520a3d28fe40a8ba55bd2f49a76dfe, a744a6e29eb9466486edc57afd937343
I20260928 16:43:53.199877 32225 raft_consensus.cc:2726] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:43:53.199846 31826 leader_election.cc:302] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e520a3d28fe40a8ba55bd2f49a76dfe, a744a6e29eb9466486edc57afd937343; no voters: 48fb050af13148fcbbbeffde7cfbf4c0
I20260928 16:43:53.200161 32224 raft_consensus.cc:2781] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:53.200569 32224 raft_consensus.cc:686] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [term 1 LEADER]: Becoming Leader. State: Replica: 3e520a3d28fe40a8ba55bd2f49a76dfe, State: Running, Role: LEADER
I20260928 16:43:53.201092 32224 consensus_queue.cc:237] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } }
I20260928 16:43:53.205652 31747 catalog_manager.cc:5495] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e520a3d28fe40a8ba55bd2f49a76dfe (127.26.101.1). New cstate: current_term: 1 leader_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e520a3d28fe40a8ba55bd2f49a76dfe" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 } health_report { overall_health: UNKNOWN } } }
I20260928 16:43:53.231935 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.229703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.26.101.1:44989)
0928 16:43:53.229878 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.229883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.230004 (+ 121us) server_negotiation.cc:407] Connection header received
0928 16:43:53.230162 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.230166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.230288 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.230377 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.231293 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.231298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.231301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.231317 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.231351 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.231363 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.231546 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.231613 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.231702 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:43:53.240835 32024 raft_consensus.cc:1261] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 1 FOLLOWER]: Refusing update from remote peer 3e520a3d28fe40a8ba55bd2f49a76dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:53.240850 32154 raft_consensus.cc:1261] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Refusing update from remote peer 3e520a3d28fe40a8ba55bd2f49a76dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:43:53.241813 32231 consensus_queue.cc:1035] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a744a6e29eb9466486edc57afd937343" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:53.242189 32224 consensus_queue.cc:1035] T 50df31bda66a4a6fb7cfb5d500dccba4 P 3e520a3d28fe40a8ba55bd2f49a76dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "48fb050af13148fcbbbeffde7cfbf4c0" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:43:53.273919 31747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:53.274267 31747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:43:53.279475 31747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e050520376434c03b52252a94dfa8caa: Sending DeleteTablet for 3 replicas of tablet 50df31bda66a4a6fb7cfb5d500dccba4
I20260928 16:43:53.280970 32003 tablet_service.cc:1514] Processing DeleteTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:53 UTC) from {username='slave'} at 127.0.0.1:47672
I20260928 16:43:53.281034 31871 tablet_service.cc:1514] Processing DeleteTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:53 UTC) from {username='slave'} at 127.0.0.1:59838
I20260928 16:43:53.281710 32134 tablet_service.cc:1514] Processing DeleteTablet for tablet 50df31bda66a4a6fb7cfb5d500dccba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:43:53 UTC) from {username='slave'} at 127.0.0.1:54436
I20260928 16:43:53.284513 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31806
I20260928 16:43:53.292985 32246 tablet_replica.cc:330] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: stopping tablet replica
I20260928 16:43:53.295292 32246 raft_consensus.cc:2229] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:53.296097 32246 raft_consensus.cc:2258] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:53.300695 32246 ts_tablet_manager.cc:1905] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:53.303066 32245 tablet_replica.cc:330] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: stopping tablet replica
I20260928 16:43:53.305052 32245 raft_consensus.cc:2229] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:43:53.308293 32246 ts_tablet_manager.cc:1918] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:53.308540 32246 log.cc:1196] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/50df31bda66a4a6fb7cfb5d500dccba4
I20260928 16:43:53.309134 32246 ts_tablet_manager.cc:1939] T 50df31bda66a4a6fb7cfb5d500dccba4 P 48fb050af13148fcbbbeffde7cfbf4c0: Deleting consensus metadata
I20260928 16:43:53.310976 31731 catalog_manager.cc:4841] TS 48fb050af13148fcbbbeffde7cfbf4c0 (127.26.101.3:40009): tablet 50df31bda66a4a6fb7cfb5d500dccba4 (table test-table [id=26bb42f9c44143378bea473390120695]) successfully deleted
I20260928 16:43:53.312099 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.285754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.26.101.1:44989)
0928 16:43:53.285949 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.285956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.302512 (+ 16556us) server_negotiation.cc:407] Connection header received
0928 16:43:53.303580 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.303587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.303655 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.303726 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.311257 (+ 7531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.311264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.311269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.311297 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.311349 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.311367 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.311475 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.311749 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.311822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:43:53.312350 32245 raft_consensus.cc:2258] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:43:53.313241 31931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:53.317962 32245 ts_tablet_manager.cc:1905] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:43:53.323380 32245 ts_tablet_manager.cc:1918] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:43:53.323632 32245 log.cc:1196] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/50df31bda66a4a6fb7cfb5d500dccba4
I20260928 16:43:53.324105 32245 ts_tablet_manager.cc:1939] T 50df31bda66a4a6fb7cfb5d500dccba4 P a744a6e29eb9466486edc57afd937343: Deleting consensus metadata
I20260928 16:43:53.325722 31734 catalog_manager.cc:4841] TS a744a6e29eb9466486edc57afd937343 (127.26.101.2:40651): tablet 50df31bda66a4a6fb7cfb5d500dccba4 (table test-table [id=26bb42f9c44143378bea473390120695]) successfully deleted
I20260928 16:43:53.408684 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31806
W20260928 16:43:53.424469 31732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.101.1:44989 (error 108)
I20260928 16:43:53.424642 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31941
I20260928 16:43:53.426867 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.426084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59870)
0928 16:43:53.426583 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.426695 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
W20260928 16:43:53.427131 31732 catalog_manager.cc:4584] TS 3e520a3d28fe40a8ba55bd2f49a76dfe (127.26.101.1:44989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50df31bda66a4a6fb7cfb5d500dccba4: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
I20260928 16:43:53.427748 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.425441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.26.101.2:40651)
0928 16:43:53.425627 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.425633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.425684 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:43:53.425892 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.425897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.425995 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.426104 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.427011 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.427017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.427022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.427042 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.427121 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.427139 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.427249 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.427409 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.427470 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:43:53.428325 32064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:53.481150 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.480617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59880)
0928 16:43:53.480862 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.480989 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 16:43:53.482254 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.481867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59882)
0928 16:43:53.482022 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.482118 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:43:53.517699 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31941
I20260928 16:43:53.533290 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32072
I20260928 16:43:53.536199 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.534135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.26.101.3:40009)
0928 16:43:53.534332 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:53.534337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:53.534355 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:43:53.534412 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:53.534417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:53.534510 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:53.534613 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:53.535464 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:53.535470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:53.535475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:53.535495 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:53.535529 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:53.535542 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:53.535660 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:53.535844 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:43:53.535897 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:43:53.536868 32195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:53.542630 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.542185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59894)
0928 16:43:53.542389 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.542483 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:43:53.543795 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.543426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59902)
0928 16:43:53.543575 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.543652 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:43:53.624624 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.624101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59918)
0928 16:43:53.624365 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.624478 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:43:53.626112 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:53.625570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44989 (local address 127.0.0.1:59920)
0928 16:43:53.625812 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:53.625925 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 16:43:53.626953 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32072
I20260928 16:43:53.642023 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 31716
I20260928 16:43:53.643332 31777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:53.750496 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 31716
2026-09-28T16:43:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2947 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 16:43:53.789491 27028 mini_oidc.cc:152] Starting JWKS server
I20260928 16:43:53.792357 27028 webserver.cc:460] Webserver started at https://127.0.0.1:34407/ using document root <none> and password file <none>
I20260928 16:43:53.792672 27028 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:43:53.793360 27028 webserver.cc:460] Webserver started at http://127.0.0.1:33003/ using document root <none> and password file <none>
2026-09-28T16:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:53Z Disabled control of system clock
I20260928 16:43:53.820513 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37399
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/testchainca.cert with env {}
W20260928 16:43:54.147644 32259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:54.147993 32259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:54.148135 32259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:54.155678 32259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:54.155833 32259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:54.155903 32259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:54.155979 32259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:54.156044 32259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:43:54.156129 32259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:43:54.156184 32259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34407/jwks.json
I20260928 16:43:54.168376 32259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:54.169756 32259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:54.171136 32259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:54.184010 32267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.183943 32264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.183976 32265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:54.184674 32259 server_base.cc:1034] running on GCE node
I20260928 16:43:54.185559 32259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:54.187141 32259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:54.188388 32259 hybrid_clock.cc:648] HybridClock initialized: now 1790613834188342 us; error 42 us; skew 500 ppm
I20260928 16:43:54.188820 32259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:54.191283 32259 webserver.cc:460] Webserver started at http://127.26.101.62:34899/ using document root <none> and password file <none>
I20260928 16:43:54.192003 32259 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:54.192116 32259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:54.192413 32259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:54.195576 32259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "e5a30bb547e24c139ef58b5ba126e3ed"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "f7612a070b9f7ebec17723b627396cac"
server_key_iv: "2984c751fa00f3ce2c4c5a6455db2d03"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.196362 32259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "e5a30bb547e24c139ef58b5ba126e3ed"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "f7612a070b9f7ebec17723b627396cac"
server_key_iv: "2984c751fa00f3ce2c4c5a6455db2d03"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.201792 32259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:54.205410 32273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:54.206990 32259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:43:54.207226 32259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "e5a30bb547e24c139ef58b5ba126e3ed"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "f7612a070b9f7ebec17723b627396cac"
server_key_iv: "2984c751fa00f3ce2c4c5a6455db2d03"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.207441 32259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:54.240631 32259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:54.241513 32259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:54.241822 32259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:54.259373 32259 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37399
I20260928 16:43:54.259405 32325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37399 every 8 connection(s)
I20260928 16:43:54.260790 32259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:54.264711 32326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:54.270591 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32259
I20260928 16:43:54.270826 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:54.271649 27028 external_mini_cluster.cc:1428] Setting key dd4b002d21b55494eb5d099c0d134686
I20260928 16:43:54.276510 32326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed: Bootstrap starting.
I20260928 16:43:54.279824 32326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:54.280843 32326 log.cc:824] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:54.284179 32326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed: No bootstrap required, opened a new log
I20260928 16:43:54.285784 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:54.273744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.26.101.62:37399)
0928 16:43:54.274822 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:54.274842 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:54.274885 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:43:54.276677 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:54.276710 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:54.277252 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:54.277634 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:54.278275 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.279647 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.281052 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.282061 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.282249 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:54.283528 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:54.283536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:54.283551 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:54.284100 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:54.284183 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:54.284205 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:54.284325 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:54.284546 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:43:54.284858 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":352,"threads_started":1}
I20260928 16:43:54.289613 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:54.291960 32326 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } }
I20260928 16:43:54.292332 32326 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:54.292469 32326 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a30bb547e24c139ef58b5ba126e3ed, State: Initialized, Role: FOLLOWER
I20260928 16:43:54.293179 32326 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } }
I20260928 16:43:54.293434 32326 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:54.293576 32326 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:54.293735 32326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:54.296018 32326 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } }
I20260928 16:43:54.296535 32326 leader_election.cc:302] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a30bb547e24c139ef58b5ba126e3ed; no voters:
I20260928 16:43:54.297079 32326 leader_election.cc:288] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:54.297240 32331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:54.297937 32331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [term 1 LEADER]: Becoming Leader. State: Replica: e5a30bb547e24c139ef58b5ba126e3ed, State: Running, Role: LEADER
I20260928 16:43:54.298513 32331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } }
I20260928 16:43:54.299348 32326 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:54.301502 32332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } } }
I20260928 16:43:54.301470 32333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a30bb547e24c139ef58b5ba126e3ed. Latest consensus state: current_term: 1 leader_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a30bb547e24c139ef58b5ba126e3ed" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37399 } } }
I20260928 16:43:54.301944 32332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:54.301944 32333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:54.303862 32338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:54.308655 32338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:54.317432 32338 catalog_manager.cc:1347] Generated new cluster ID: 524eaf33893844b8a3196cdf855497bc
I20260928 16:43:54.317564 32338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:54.329136 32338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:54.330436 32338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:54.344056 32338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a30bb547e24c139ef58b5ba126e3ed: Generated new TSK 0
I20260928 16:43:54.344972 32338 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:43:54.381211 32356 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 16:43:54.381402 32356 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:43:54.383742 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:54.360635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.26.101.62:37399)
0928 16:43:54.360827 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:54.360834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:54.361301 (+ 467us) server_negotiation.cc:407] Connection header received
0928 16:43:54.361664 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:54.361669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:54.361752 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:54.361889 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:43:54.362483 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.363301 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.364757 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.365215 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.365330 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:54.365965 (+ 635us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:43:54.383194 (+ 17229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:43:54.383520 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:43:54.383862 32329 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 16:43:54.385738 32355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:54.360552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37399 (local address 127.0.0.1:56402)
0928 16:43:54.361185 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:54.361202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:54.361393 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:54.361949 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:54.361954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:54.361965 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:43:54.362305 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:54.362315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.363449 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.363453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:54.364599 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:54.364612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:54.365430 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:54.365435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:54.365575 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:54.365644 (+ 69us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:43:54.383558 (+ 17914us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:43:54.383943 (+ 385us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.101.62:37399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":325,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.26.101.62:37399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 16:43:54.387670 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32259
I20260928 16:43:54.388888 32321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:54.483750 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32259
2026-09-28T16:43:54Z chronyd exiting
I20260928 16:43:54.532549 27028 test_util.cc:175] -----------------------------------------------
I20260928 16:43:54.532696 27028 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613789516984-27028-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 16:43:54.536903 27028 mini_oidc.cc:152] Starting JWKS server
I20260928 16:43:54.538694 27028 webserver.cc:460] Webserver started at https://127.0.0.1:35921/ using document root <none> and password file <none>
I20260928 16:43:54.538892 27028 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:43:54.539616 27028 webserver.cc:460] Webserver started at http://127.0.0.1:32819/ using document root <none> and password file <none>
2026-09-28T16:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:54Z Disabled control of system clock
I20260928 16:43:54.566690 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44377
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/testchainca.cert with env {}
W20260928 16:43:54.891748 32363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:54.892068 32363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:54.892158 32363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:54.899552 32363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:54.899689 32363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:54.899726 32363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:54.899781 32363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:54.899823 32363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:43:54.899884 32363 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:43:54.899921 32363 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35921/jwks.json
I20260928 16:43:54.910132 32363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:54.911535 32363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:54.912863 32363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:54.924750 32368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.924682 32371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:54.924728 32369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:54.925318 32363 server_base.cc:1034] running on GCE node
I20260928 16:43:54.926162 32363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:54.927726 32363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:54.928960 32363 hybrid_clock.cc:648] HybridClock initialized: now 1790613834928901 us; error 61 us; skew 500 ppm
I20260928 16:43:54.929335 32363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:54.936990 32363 webserver.cc:460] Webserver started at http://127.26.101.62:38703/ using document root <none> and password file <none>
I20260928 16:43:54.937768 32363 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:54.937888 32363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:54.938220 32363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:54.941406 32363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "8fc71e552385489c87438495ca500391"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "4c9ccf7a3073f7d95173c5ba6021abd2"
server_key_iv: "d3d6e77422517cc01d5f1b5aaccba96c"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.942225 32363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "8fc71e552385489c87438495ca500391"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "4c9ccf7a3073f7d95173c5ba6021abd2"
server_key_iv: "d3d6e77422517cc01d5f1b5aaccba96c"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.947819 32363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:43:54.951563 32377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:54.953202 32363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:43:54.953420 32363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8fc71e552385489c87438495ca500391"
format_stamp: "Formatted at 2026-09-28 16:43:54 on dist-test-slave-t0tw"
server_key: "4c9ccf7a3073f7d95173c5ba6021abd2"
server_key_iv: "d3d6e77422517cc01d5f1b5aaccba96c"
server_key_version: "encryptionkey@0"
I20260928 16:43:54.953709 32363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:54.980962 32363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:54.981886 32363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:54.982196 32363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:55.000399 32363 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:44377
I20260928 16:43:55.000422 32428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:44377 every 8 connection(s)
I20260928 16:43:55.001832 32363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:55.005854 32429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:55.007298 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32363
I20260928 16:43:55.007465 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:55.008111 27028 external_mini_cluster.cc:1428] Setting key 66b6e5501a59ddf37b59ef904a0b81f8
I20260928 16:43:55.018858 32429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: Bootstrap starting.
I20260928 16:43:55.021418 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.010058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.26.101.62:44377)
0928 16:43:55.011239 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.011262 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.011315 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:43:55.012875 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.012899 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.013389 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.013765 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:55.014335 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.015681 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.016926 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.017600 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.017762 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.018995 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:55.019004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:55.019023 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:55.019688 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:55.019770 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:55.019791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:55.019890 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:55.020111 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:43:55.020440 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":450,"threads_started":1}
I20260928 16:43:55.022374 32429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:55.023555 32429 log.cc:824] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:55.025394 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:55.026427 32429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: No bootstrap required, opened a new log
I20260928 16:43:55.034245 32429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } }
I20260928 16:43:55.034631 32429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:55.034734 32429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc71e552385489c87438495ca500391, State: Initialized, Role: FOLLOWER
I20260928 16:43:55.035374 32429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } }
I20260928 16:43:55.035689 32429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:55.035832 32429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:55.036024 32429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:55.038208 32429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } }
I20260928 16:43:55.038728 32429 leader_election.cc:302] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fc71e552385489c87438495ca500391; no voters:
I20260928 16:43:55.039322 32429 leader_election.cc:288] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:55.039474 32434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:55.040160 32434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [term 1 LEADER]: Becoming Leader. State: Replica: 8fc71e552385489c87438495ca500391, State: Running, Role: LEADER
I20260928 16:43:55.040766 32434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } }
I20260928 16:43:55.041466 32429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:55.043699 32435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fc71e552385489c87438495ca500391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } } }
I20260928 16:43:55.043723 32436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fc71e552385489c87438495ca500391. Latest consensus state: current_term: 1 leader_uuid: "8fc71e552385489c87438495ca500391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fc71e552385489c87438495ca500391" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44377 } } }
I20260928 16:43:55.044294 32435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:55.044301 32436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391 [sys.catalog]: This master's current role is: LEADER
W20260928 16:43:55.051995 32450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:43:55.052142 32450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:43:55.052286 32441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:55.054131 32441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:55.062919 32441 catalog_manager.cc:1347] Generated new cluster ID: 92ab0b129ae7465f94fdb4a577bb0160
I20260928 16:43:55.063040 32441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:55.080278 32441 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:55.082181 32441 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:55.094086 32441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fc71e552385489c87438495ca500391: Generated new TSK 0
I20260928 16:43:55.095014 32441 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:43:55.145988 32459 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 16:43:55.146139 32459 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:43:55.147213 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.118763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.26.101.62:44377)
0928 16:43:55.118994 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.119001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.119657 (+ 656us) server_negotiation.cc:407] Connection header received
0928 16:43:55.119974 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.119979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.120056 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.120193 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:43:55.120815 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.121620 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.123697 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.124190 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.124316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.124972 (+ 656us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:43:55.146559 (+ 21587us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:43:55.146928 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 16:43:55.147344 32432 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 16:43:55.147567 32458 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:55.118571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44377 (local address 127.0.0.1:34600)
0928 16:43:55.119539 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:55.119560 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:55.119723 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:55.120232 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:55.120237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:55.120251 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:43:55.120614 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:55.120628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.121771 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.121777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:55.123514 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:55.123532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.124375 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.124383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:55.124601 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.124652 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:43:55.146827 (+ 22175us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:43:55.147156 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.101.62:44377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":657,"threads_started":1}
I20260928 16:43:55.148708 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32363
I20260928 16:43:55.149964 32424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:55.244398 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32363
2026-09-28T16:43:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 16:43:55.298260 27028 mini_oidc.cc:152] Starting JWKS server
I20260928 16:43:55.300212 27028 webserver.cc:460] Webserver started at https://127.0.0.1:33333/ using document root <none> and password file <none>
I20260928 16:43:55.300490 27028 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:43:55.301189 27028 webserver.cc:460] Webserver started at http://127.0.0.1:35223/ using document root <none> and password file <none>
2026-09-28T16:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:55Z Disabled control of system clock
I20260928 16:43:55.327630 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37659
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37659
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33333/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:43:55.647392 32467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:55.647725 32467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:55.647868 32467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:55.655766 32467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:55.655901 32467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:55.655969 32467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:55.656049 32467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:55.656111 32467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:43:55.656191 32467 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:43:55.656244 32467 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33333/jwks.json
I20260928 16:43:55.666191 32467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33333/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:55.667667 32467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:55.669585 32467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:55.681802 32473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:55.681746 32475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:55.681838 32472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:55.683182 32467 server_base.cc:1034] running on GCE node
I20260928 16:43:55.684055 32467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:55.685781 32467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:55.687023 32467 hybrid_clock.cc:648] HybridClock initialized: now 1790613835686958 us; error 61 us; skew 500 ppm
I20260928 16:43:55.687569 32467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:55.690049 32467 webserver.cc:460] Webserver started at http://127.26.101.62:33487/ using document root <none> and password file <none>
I20260928 16:43:55.690814 32467 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:55.690912 32467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:55.691303 32467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:55.694345 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "accdc4bf0326443f998aea17425df3f6"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-t0tw"
server_key: "979fb3698f979788699fd279edba23c0"
server_key_iv: "348e66f0644932fc816ace1375a7e08e"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.695191 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "accdc4bf0326443f998aea17425df3f6"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-t0tw"
server_key: "979fb3698f979788699fd279edba23c0"
server_key_iv: "348e66f0644932fc816ace1375a7e08e"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.700560 32467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 16:43:55.704475 32481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:55.705974 32467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:55.706152 32467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "accdc4bf0326443f998aea17425df3f6"
format_stamp: "Formatted at 2026-09-28 16:43:55 on dist-test-slave-t0tw"
server_key: "979fb3698f979788699fd279edba23c0"
server_key_iv: "348e66f0644932fc816ace1375a7e08e"
server_key_version: "encryptionkey@0"
I20260928 16:43:55.706360 32467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:55.737952 32467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:55.738833 32467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:55.739164 32467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:55.756973 32467 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37659
I20260928 16:43:55.756980 32532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37659 every 8 connection(s)
I20260928 16:43:55.758359 32467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:55.762200 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:55.766909 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32467
I20260928 16:43:55.767155 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:55.767937 27028 external_mini_cluster.cc:1428] Setting key bdb59943a5bdbda243b5f853c79009ea
I20260928 16:43:55.773931 32533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6: Bootstrap starting.
I20260928 16:43:55.777005 32533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:55.778035 32533 log.cc:824] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:55.781011 32533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6: No bootstrap required, opened a new log
I20260928 16:43:55.781054 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.769897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.26.101.62:37659)
0928 16:43:55.770917 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.770932 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.770976 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:43:55.772705 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.772730 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.773187 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.773555 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:55.774132 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.775457 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.776758 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.777474 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.777663 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.778790 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:55.778797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:55.778811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:55.779386 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:55.779469 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:55.779490 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:55.779602 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:55.779821 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:43:55.780118 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":338,"threads_started":1}
I20260928 16:43:55.785104 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:55.789117 32533 raft_consensus.cc:348] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } }
I20260928 16:43:55.789489 32533 raft_consensus.cc:374] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:55.789594 32533 raft_consensus.cc:729] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accdc4bf0326443f998aea17425df3f6, State: Initialized, Role: FOLLOWER
I20260928 16:43:55.790122 32533 consensus_queue.cc:260] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } }
I20260928 16:43:55.790341 32533 raft_consensus.cc:388] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:55.790472 32533 raft_consensus.cc:482] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:55.790637 32533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:55.793023 32533 raft_consensus.cc:504] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } }
I20260928 16:43:55.793491 32533 leader_election.cc:302] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: accdc4bf0326443f998aea17425df3f6; no voters:
I20260928 16:43:55.794050 32533 leader_election.cc:288] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:55.794195 32538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:55.794895 32538 raft_consensus.cc:686] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [term 1 LEADER]: Becoming Leader. State: Replica: accdc4bf0326443f998aea17425df3f6, State: Running, Role: LEADER
I20260928 16:43:55.795490 32538 consensus_queue.cc:237] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } }
I20260928 16:43:55.796241 32533 sys_catalog.cc:564] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:55.798514 32540 sys_catalog.cc:455] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader accdc4bf0326443f998aea17425df3f6. Latest consensus state: current_term: 1 leader_uuid: "accdc4bf0326443f998aea17425df3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } } }
I20260928 16:43:55.798491 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "accdc4bf0326443f998aea17425df3f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accdc4bf0326443f998aea17425df3f6" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37659 } } }
I20260928 16:43:55.798934 32540 sys_catalog.cc:458] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:55.798940 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:55.800490 32545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:55.806375 32545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:55.817068 32545 catalog_manager.cc:1347] Generated new cluster ID: 9a12b29af815434abff388f82e527f4e
I20260928 16:43:55.817200 32545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:55.830473 32545 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:55.832402 32545 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:55.843736 32545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P accdc4bf0326443f998aea17425df3f6: Generated new TSK 0
I20260928 16:43:55.844925 32545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:55.867522 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:55.858924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.26.101.62:37659)
0928 16:43:55.859145 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:55.859159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:55.859521 (+ 362us) server_negotiation.cc:407] Connection header received
0928 16:43:55.859731 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:55.859736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:55.859836 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:55.860006 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:43:55.860461 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.861583 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.863023 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.863738 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.863867 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.864361 (+ 494us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:43:55.866805 (+ 2444us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:43:55.867204 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:43:55.867660 32536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 16:43:55.870172 32562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:55.858825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37659 (local address 127.0.0.1:37764)
0928 16:43:55.859420 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:55.859436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:55.859563 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:55.860016 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:55.860020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:55.860030 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:43:55.860297 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:55.860305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.861773 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.861778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:55.862877 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:55.862890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:55.863914 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:55.863918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:55.864044 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:55.864078 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:43:55.867039 (+ 2961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:43:55.869756 (+ 2717us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.101.62:37659: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":2360,"thread_start_us":331,"threads_started":1}
I20260928 16:43:55.871229 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32467
I20260928 16:43:55.872401 32528 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 16:43:55.879719 32563 webserver.cc:533] Webserver: sslize failed:
E20260928 16:43:55.879825 32563 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:43:55.963917 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32467
2026-09-28T16:43:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T16:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:56Z Disabled control of system clock
I20260928 16:43:56.042946 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:38509
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:38509
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/testchainca.cert with env {}
W20260928 16:43:56.368075 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:56.368372 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:56.368458 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:56.375572 32568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:56.375682 32568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:56.375720 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:56.375778 32568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:56.375821 32568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:43:56.375880 32568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:43:56.375919 32568 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 16:43:56.385934 32568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:38509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:56.387233 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:56.388522 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:56.400172 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:56.400278 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:56.400197 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:56.401134 32568 server_base.cc:1034] running on GCE node
I20260928 16:43:56.401913 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:56.403350 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:56.404594 32568 hybrid_clock.cc:648] HybridClock initialized: now 1790613836404523 us; error 63 us; skew 500 ppm
I20260928 16:43:56.404927 32568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:43:56.407212 32568 webserver.cc:460] Webserver started at http://127.26.101.62:34445/ using document root <none> and password file <none>
I20260928 16:43:56.407905 32568 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:56.408012 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:56.408306 32568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:56.411298 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "932eceb3dab54dc8a18bdd764ed36252"
format_stamp: "Formatted at 2026-09-28 16:43:56 on dist-test-slave-t0tw"
server_key: "5666d6751ea2a878c0e9b470bf39c4c1"
server_key_iv: "52036454810d960139e170866f880868"
server_key_version: "encryptionkey@0"
I20260928 16:43:56.412101 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "932eceb3dab54dc8a18bdd764ed36252"
format_stamp: "Formatted at 2026-09-28 16:43:56 on dist-test-slave-t0tw"
server_key: "5666d6751ea2a878c0e9b470bf39c4c1"
server_key_iv: "52036454810d960139e170866f880868"
server_key_version: "encryptionkey@0"
I20260928 16:43:56.417121 32568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:43:56.420539 32582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:56.421921 32568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:43:56.422088 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "932eceb3dab54dc8a18bdd764ed36252"
format_stamp: "Formatted at 2026-09-28 16:43:56 on dist-test-slave-t0tw"
server_key: "5666d6751ea2a878c0e9b470bf39c4c1"
server_key_iv: "52036454810d960139e170866f880868"
server_key_version: "encryptionkey@0"
I20260928 16:43:56.422291 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:56.440410 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:56.441313 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:56.441588 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:56.459023 32568 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:38509
I20260928 16:43:56.459044 32633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:38509 every 8 connection(s)
I20260928 16:43:56.460454 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:56.461596 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32568
I20260928 16:43:56.461779 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:56.462417 27028 external_mini_cluster.cc:1428] Setting key 7c4cfc5f34888252eac39e5a9513eeeb
I20260928 16:43:56.464881 32634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:43:56.476194 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:56.464378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.26.101.62:38509)
0928 16:43:56.465400 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:56.465412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:56.465457 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:43:56.467041 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:56.467064 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:56.467613 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:56.467982 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:56.468593 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.469931 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.471751 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.472423 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.472567 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:56.473822 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:56.473830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:56.473848 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:43:56.474449 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:56.474529 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:43:56.474554 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:56.474645 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:56.474865 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 16:43:56.475237 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":338,"threads_started":1}
I20260928 16:43:56.477262 32634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252: Bootstrap starting.
I20260928 16:43:56.480005 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:43:56.480585 32634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:56.481545 32634 log.cc:824] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:56.484544 32634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252: No bootstrap required, opened a new log
I20260928 16:43:56.492213 32634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } }
I20260928 16:43:56.492578 32634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:56.492661 32634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932eceb3dab54dc8a18bdd764ed36252, State: Initialized, Role: FOLLOWER
I20260928 16:43:56.493187 32634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } }
I20260928 16:43:56.493396 32634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:56.493492 32634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:56.493614 32634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:56.496049 32634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } }
I20260928 16:43:56.496528 32634 leader_election.cc:302] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932eceb3dab54dc8a18bdd764ed36252; no voters:
I20260928 16:43:56.497083 32634 leader_election.cc:288] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:56.497223 32639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:56.497911 32639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [term 1 LEADER]: Becoming Leader. State: Replica: 932eceb3dab54dc8a18bdd764ed36252, State: Running, Role: LEADER
I20260928 16:43:56.498488 32639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } }
I20260928 16:43:56.499118 32634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:56.501478 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932eceb3dab54dc8a18bdd764ed36252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } } }
I20260928 16:43:56.501855 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:56.501576 32642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932eceb3dab54dc8a18bdd764ed36252. Latest consensus state: current_term: 1 leader_uuid: "932eceb3dab54dc8a18bdd764ed36252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932eceb3dab54dc8a18bdd764ed36252" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38509 } } }
I20260928 16:43:56.502192 32642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:56.503420 32646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:56.508810 32646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:56.517894 32646 catalog_manager.cc:1347] Generated new cluster ID: 2303d6360ebe4698beeb9b29b87e2b59
I20260928 16:43:56.518052 32646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:56.541316 32646 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:56.543192 32646 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:56.551741 32646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932eceb3dab54dc8a18bdd764ed36252: Generated new TSK 0
I20260928 16:43:56.552783 32646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:43:56.629280 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:56.569728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.26.101.62:38509)
0928 16:43:56.569967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:56.569974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:56.570364 (+ 390us) server_negotiation.cc:407] Connection header received
0928 16:43:56.570621 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:56.570625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:56.570704 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:56.570835 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:43:56.571373 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.572111 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.574160 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.574638 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.574741 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:56.575392 (+ 651us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:43:56.628651 (+ 53259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:43:56.629020 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50712: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 16:43:56.629431 32637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50712: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 16:43:56.629688 32664 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:43:56.569617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:38509 (local address 127.0.0.1:50712)
0928 16:43:56.570246 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:56.570266 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:56.570433 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:56.570872 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:56.570877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:56.570889 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:43:56.571220 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:56.571233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.572254 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.572260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:56.573957 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:56.573974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:56.574805 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:56.574812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:56.575017 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:56.575064 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:43:56.628961 (+ 53897us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:43:56.629276 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.26.101.62:38509: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":314,"threads_started":1}
I20260928 16:43:56.630765 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32568
I20260928 16:43:56.631974 32629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:43:56.729892 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32568
2026-09-28T16:43:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (745 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (416 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:43:57 dist-test-slave-t0tw krb5kdc[32680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:43:57 dist-test-slave-t0tw krb5kdc[32680](info): set up 2 sockets
Sep 28 16:43:57 dist-test-slave-t0tw krb5kdc[32680](info): commencing operation
krb5kdc: starting...
W20260928 16:43:58.575446 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:43:58 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:43:58.786864 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37625
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:59.106472 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:59.106802 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:59.106920 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:59.114372 32693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:43:59.114508 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:59.114565 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:59.114646 32693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:43:59.114701 32693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:43:59.126127 32693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:59.127622 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:59.128952 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:59.140749 32698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.140815 32701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.140782 32699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:59.141749 32693 server_base.cc:1034] running on GCE node
I20260928 16:43:59.142642 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:59.144172 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:59.145383 32693 hybrid_clock.cc:648] HybridClock initialized: now 1790613839145322 us; error 53 us; skew 500 ppm
I20260928 16:43:59.145744 32693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:59 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613839, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:59.148921 32693 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:43:59.150446 32693 webserver.cc:460] Webserver started at http://127.26.101.62:38353/ using document root <none> and password file <none>
I20260928 16:43:59.151196 32693 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:59.151309 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:59.151594 32693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:59.154585 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "e2451b5745ee47b39663694039304731"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "f213524e6e96a35e534a7b307c9720bb"
server_key_iv: "cf457a2371ff38a392e0587743d1c1f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.155479 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "e2451b5745ee47b39663694039304731"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "f213524e6e96a35e534a7b307c9720bb"
server_key_iv: "cf457a2371ff38a392e0587743d1c1f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.160779 32693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:43:59.164599 32708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:59.166132 32693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:43:59.166325 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "e2451b5745ee47b39663694039304731"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "f213524e6e96a35e534a7b307c9720bb"
server_key_iv: "cf457a2371ff38a392e0587743d1c1f0"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.166536 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:59.191870 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:59.196569 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:59.196861 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:59.214639 32693 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37625
I20260928 16:43:59.214672 32760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37625 every 8 connection(s)
I20260928 16:43:59.216110 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:43:59.216848 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 32693
I20260928 16:43:59.217079 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:43:59.217857 27028 external_mini_cluster.cc:1428] Setting key d839786444bc89747960511a56bd0a91
I20260928 16:43:59.221065 32761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:43:59 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:59.232339 32761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731: Bootstrap starting.
I20260928 16:43:59.235855 32761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731: Neither blocks nor log segments found. Creating new log.
I20260928 16:43:59.236941 32761 log.cc:824] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731: Log is configured to *not* fsync() on all Append() calls
I20260928 16:43:59.240463 32761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731: No bootstrap required, opened a new log
I20260928 16:43:59.241528 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:59.220054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.26.101.62:37625)
0928 16:43:59.220932 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:59.220950 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:59.221023 (+ 73us) server_negotiation.cc:407] Connection header received
0928 16:43:59.222927 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:59.222951 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:59.223403 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:59.223805 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:59.225276 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.226674 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.227972 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.228673 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.228861 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:59.233294 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:59.233302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:59.233318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:59.233364 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:59.236757 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:59.237582 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:59.237595 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:59.237604 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:59.237800 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:59.238242 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:59.238246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:59.238249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:59.238937 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:59.239264 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:59.239646 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 16:43:59.240131 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":350,"threads_started":1}
I20260928 16:43:59.248922 32761 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } }
I20260928 16:43:59.249348 32761 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:43:59.249428 32761 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2451b5745ee47b39663694039304731, State: Initialized, Role: FOLLOWER
I20260928 16:43:59.249959 32761 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } }
I20260928 16:43:59.250157 32761 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:43:59.250253 32761 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:43:59.250378 32761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:43:59.252514 32761 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } }
I20260928 16:43:59.253094 32761 leader_election.cc:302] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2451b5745ee47b39663694039304731; no voters:
I20260928 16:43:59.253648 32761 leader_election.cc:288] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:43:59.253777 32766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:43:59.254416 32766 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [term 1 LEADER]: Becoming Leader. State: Replica: e2451b5745ee47b39663694039304731, State: Running, Role: LEADER
I20260928 16:43:59.254972 32766 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } }
I20260928 16:43:59.255720 32761 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:43:59.257771 300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2451b5745ee47b39663694039304731. Latest consensus state: current_term: 1 leader_uuid: "e2451b5745ee47b39663694039304731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } } }
I20260928 16:43:59.257799 32767 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2451b5745ee47b39663694039304731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2451b5745ee47b39663694039304731" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37625 } } }
I20260928 16:43:59.258256 32767 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:59.258317 300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731 [sys.catalog]: This master's current role is: LEADER
I20260928 16:43:59.259644 303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:43:59.265987 303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:43:59.275720 303 catalog_manager.cc:1347] Generated new cluster ID: da2c710e38fb4c75bd7fea745345c6fb
I20260928 16:43:59.275859 303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:43:59.328806 303 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:43:59.330736 303 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:43:59.343160 303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2451b5745ee47b39663694039304731: Generated new TSK 0
I20260928 16:43:59.344280 303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:43:59.407642 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:43:59.749058 319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:43:59.749356 319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:43:59.749459 319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:43:59.756830 319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:43:59.756949 319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:43:59.757100 319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:43:59.767064 319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:43:59.768445 319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:43:59.769855 319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:43:59.783591 327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.783612 325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:43:59.783828 324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:43:59.784518 319 server_base.cc:1034] running on GCE node
I20260928 16:43:59.785132 319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:43:59.786119 319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:43:59.787322 319 hybrid_clock.cc:648] HybridClock initialized: now 1790613839787277 us; error 34 us; skew 500 ppm
I20260928 16:43:59.787674 319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:43:59 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613839, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:43:59.790879 319 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:43:59.792431 319 webserver.cc:460] Webserver started at http://127.26.101.1:43711/ using document root <none> and password file <none>
I20260928 16:43:59.793160 319 fs_manager.cc:359] Metadata directory not provided
I20260928 16:43:59.793286 319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:43:59.793593 319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:43:59.796587 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "7dfcd610cc744fabb9676633878c3128"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "79e5f23f34f4123e80361a52441b2e38"
server_key_iv: "c1843f7c5311d54eef046e8084220132"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.797361 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "7dfcd610cc744fabb9676633878c3128"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "79e5f23f34f4123e80361a52441b2e38"
server_key_iv: "c1843f7c5311d54eef046e8084220132"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.802661 319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:43:59.806389 334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:59.807880 319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:43:59.808055 319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "7dfcd610cc744fabb9676633878c3128"
format_stamp: "Formatted at 2026-09-28 16:43:59 on dist-test-slave-t0tw"
server_key: "79e5f23f34f4123e80361a52441b2e38"
server_key_iv: "c1843f7c5311d54eef046e8084220132"
server_key_version: "encryptionkey@0"
I20260928 16:43:59.808255 319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:43:59.828984 319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:43:59.833384 319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:43:59.833673 319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:43:59.834693 319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:43:59.836118 319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:43:59.836200 319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:59.836319 319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:43:59.836372 319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:43:59.868268 319 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:32987
I20260928 16:43:59.868321 449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:32987 every 8 connection(s)
I20260928 16:43:59.869660 319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:43:59.877684 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 319
I20260928 16:43:59.877884 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:43:59.878500 27028 external_mini_cluster.cc:1428] Setting key 53cfd8151ede3814aa1c30786e310412
Sep 28 16:43:59 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613839, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:43:59.890725 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:59.872560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:42403 (local address 127.26.101.62:37625)
0928 16:43:59.873014 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 16:43:59.873019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:43:59.874207 (+ 1188us) server_negotiation.cc:407] Connection header received
0928 16:43:59.876129 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:43:59.876133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:43:59.876205 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:43:59.876309 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:43:59.877909 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.878665 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.880481 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.880885 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.881006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:59.884590 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:43:59.884596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:43:59.884599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:43:59.884633 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:43:59.887364 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:59.888210 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:59.888215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:59.888217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:59.888350 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:43:59.888849 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:43:59.888852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:43:59.888854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:43:59.889152 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:43:59.889291 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:43:59.890114 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 16:43:59.890359 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260928 16:43:59.891853 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:43:59.873241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37625 (local address 127.26.101.1:42403)
0928 16:43:59.874028 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 16:43:59.874068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:43:59.875856 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:43:59.876580 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:43:59.876589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:43:59.877116 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:43:59.877704 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:59.877723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.878824 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.878828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:59.880339 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:43:59.880352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:43:59.881112 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:43:59.881119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:43:59.881348 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:43:59.882058 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:43:59.882088 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:43:59.884347 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:43:59.887571 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:59.887579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:59.887595 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:59.888067 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:59.888469 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:43:59.888474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:43:59.888476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:43:59.888718 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:43:59.889282 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:43:59.889291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:43:59.889671 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 16:43:59.889958 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 16:43:59.890573 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":307,"threads_started":1}
I20260928 16:43:59.893504 450 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37625
I20260928 16:43:59.893800 450 heartbeater.cc:464] Registering TS with master...
I20260928 16:43:59.894536 450 heartbeater.cc:511] Master 127.26.101.62:37625 requested a full tablet report, sending...
I20260928 16:43:59.897151 32725 ts_manager.cc:194] Registered new tserver with Master: 7dfcd610cc744fabb9676633878c3128 (127.26.101.1:32987)
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:43:59.898779 32725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:42403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:43:59.926916 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:00.261904 456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:00.262244 456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:00.262384 456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:00.270025 456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:00.270169 456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:00.270334 456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:00.280678 456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:00.282099 456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:00.283475 456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:00.297175 462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:00.297139 461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:00.297793 464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:00.298977 456 server_base.cc:1034] running on GCE node
I20260928 16:44:00.299726 456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:00.300806 456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:00.302045 456 hybrid_clock.cc:648] HybridClock initialized: now 1790613840301990 us; error 46 us; skew 500 ppm
I20260928 16:44:00.302397 456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:00.305933 456 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:44:00.307566 456 webserver.cc:460] Webserver started at http://127.26.101.2:42867/ using document root <none> and password file <none>
I20260928 16:44:00.308313 456 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:00.308427 456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:00.308699 456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:00.311842 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "8775d5b6fe1e41109a70006dce4d1877"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "6382d03ff6bea0d0c8f3b9e987caed3d"
server_key_iv: "62f2a07f02aa751964ec03338d8783a4"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.312680 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "8775d5b6fe1e41109a70006dce4d1877"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "6382d03ff6bea0d0c8f3b9e987caed3d"
server_key_iv: "62f2a07f02aa751964ec03338d8783a4"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.318264 456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:00.322175 471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.323733 456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:44:00.323908 456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "8775d5b6fe1e41109a70006dce4d1877"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "6382d03ff6bea0d0c8f3b9e987caed3d"
server_key_iv: "62f2a07f02aa751964ec03338d8783a4"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.324100 456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:00.353770 456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:00.358314 456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:00.358582 456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:00.359583 456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:00.360989 456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:00.361069 456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.361167 456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:00.361212 456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.392822 456 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:44543
I20260928 16:44:00.392911 585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:44543 every 8 connection(s)
I20260928 16:44:00.394203 456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:00.398406 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 456
I20260928 16:44:00.398675 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:00.399286 27028 external_mini_cluster.cc:1428] Setting key 49a8fa15dc948afae2d993c3ade0c717
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:00.417906 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:00.396921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:39385 (local address 127.26.101.62:37625)
0928 16:44:00.397394 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:00.397400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:00.398633 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 16:44:00.400948 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:00.400953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:00.401041 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:00.401141 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:00.403191 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.403983 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.406157 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.406686 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.406808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:00.411295 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:00.411303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:00.411306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:00.411345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:00.414215 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.415256 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.415261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.415263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.415412 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.415944 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.415947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.415949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.416232 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:00.416366 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:00.417324 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 16:44:00.417568 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260928 16:44:00.419345 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:00.397541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37625 (local address 127.26.101.2:39385)
0928 16:44:00.398373 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:00.398431 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:00.400612 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:00.401401 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:00.401410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:00.402146 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:00.402982 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:00.403010 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.404146 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.404150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:00.405956 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:00.405991 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.406933 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.406940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:00.407274 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:00.408038 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:00.408070 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:00.410957 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:00.414398 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:00.414409 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:00.414430 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:00.415103 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:00.415567 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:00.415572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:00.415575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:00.415839 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:00.416353 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:00.416362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:00.416856 (+ 494us) client_negotiation.cc:769] Sending connection context
0928 16:44:00.417187 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 16:44:00.417793 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":292,"threads_started":1}
I20260928 16:44:00.420855 586 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37625
I20260928 16:44:00.421175 586 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:44:00.421967 586 heartbeater.cc:511] Master 127.26.101.62:37625 requested a full tablet report, sending...
I20260928 16:44:00.423403 32725 ts_manager.cc:194] Registered new tserver with Master: 8775d5b6fe1e41109a70006dce4d1877 (127.26.101.2:44543)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:44:00.424203 32725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:39385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:44:00.449908 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:00.779994 592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:00.780313 592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:00.780479 592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:00.787884 592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:00.788034 592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:00.788190 592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:00.798307 592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:00.799966 592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:00.801450 592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:00.815057 600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:00.815104 598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:00.815332 597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:00.815845 592 server_base.cc:1034] running on GCE node
I20260928 16:44:00.816447 592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:00.817447 592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:00.818658 592 hybrid_clock.cc:648] HybridClock initialized: now 1790613840818609 us; error 41 us; skew 500 ppm
I20260928 16:44:00.819011 592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:00.822310 592 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:44:00.823830 592 webserver.cc:460] Webserver started at http://127.26.101.3:39149/ using document root <none> and password file <none>
I20260928 16:44:00.824523 592 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:00.824640 592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:00.824947 592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:00.828090 592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "dbb84f2fdc624c4ebcb5ad2c87b3e710"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "2b54926c63317ef132bac8a8569980bc"
server_key_iv: "de2805f1440e976271ee856a97b3a5fb"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.828965 592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "dbb84f2fdc624c4ebcb5ad2c87b3e710"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "2b54926c63317ef132bac8a8569980bc"
server_key_iv: "de2805f1440e976271ee856a97b3a5fb"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.834546 592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:00.838382 607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.839910 592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:00.840091 592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "dbb84f2fdc624c4ebcb5ad2c87b3e710"
format_stamp: "Formatted at 2026-09-28 16:44:00 on dist-test-slave-t0tw"
server_key: "2b54926c63317ef132bac8a8569980bc"
server_key_iv: "de2805f1440e976271ee856a97b3a5fb"
server_key_version: "encryptionkey@0"
I20260928 16:44:00.840294 592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:00.861199 592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:00.865851 592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:00.866184 592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:00.867304 592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:00.868875 592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:00.868960 592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.869065 592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:00.869117 592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:00.901007 592 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:41747
I20260928 16:44:00.901053 721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:41747 every 8 connection(s)
I20260928 16:44:00.902050 450 heartbeater.cc:503] Master 127.26.101.62:37625 was elected leader, sending a full tablet report...
I20260928 16:44:00.902346 592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:00.910897 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 592
I20260928 16:44:00.911057 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:00.911504 27028 external_mini_cluster.cc:1428] Setting key 017eb846491b54db1890e2827cb3aa96
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:00.925189 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:00.905289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:49277 (local address 127.26.101.62:37625)
0928 16:44:00.905507 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:00.905511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:00.906670 (+ 1159us) server_negotiation.cc:407] Connection header received
0928 16:44:00.908668 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:00.908673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:00.908762 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:00.908852 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:00.910866 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.911866 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.914314 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.915003 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.915199 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:00.919233 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:00.919238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:00.919242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:00.919278 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:00.921920 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.922736 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.922740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.922742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.922862 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.923426 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.923432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.923435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.923764 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:00.923924 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:00.924721 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 16:44:00.924922 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:44:00.926479 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:00.905736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37625 (local address 127.26.101.3:49277)
0928 16:44:00.906508 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:00.906550 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:00.908364 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:00.909150 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:00.909160 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:00.909724 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:00.910580 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:00.910615 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.912078 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.912082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:00.914119 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:00.914136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.915340 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.915349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:00.915661 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:00.916500 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:00.916532 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:00.918905 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:00.922097 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:00.922105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:00.922122 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:00.922629 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:00.923002 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:00.923006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:00.923009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:00.923282 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:00.923939 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:00.923946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:00.924321 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 16:44:00.924598 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 16:44:00.925048 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":302,"threads_started":1}
I20260928 16:44:00.928048 722 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37625
I20260928 16:44:00.928439 722 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:00.929219 722 heartbeater.cc:511] Master 127.26.101.62:37625 requested a full tablet report, sending...
I20260928 16:44:00.930783 32725 ts_manager.cc:194] Registered new tserver with Master: dbb84f2fdc624c4ebcb5ad2c87b3e710 (127.26.101.3:41747)
I20260928 16:44:00.931705 32725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:49277
I20260928 16:44:00.941284 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/test-admin.keytab.
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:00.968830 27028 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 16:44:00.974413 729 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:00.972001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.26.101.1:32987)
0928 16:44:00.972647 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:00.972654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:00.972681 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:44:00.972744 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:00.972751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:00.973000 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:00.973141 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:00.973993 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33974: BlockingRecv error: recv got EOF from 127.0.0.1:33974 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":334,"threads_started":1}
W20260928 16:44:00.974494 27028 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:00.976387 27028 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:00.977859 27028 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:00 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:44:00.995649 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:00.979669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.26.101.1:32987)
0928 16:44:00.979883 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:00.979890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:00.980139 (+ 249us) server_negotiation.cc:407] Connection header received
0928 16:44:00.980278 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:00.980282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:00.980353 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:00.980461 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:00.981607 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.982650 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.983989 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:00.984616 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:00.984742 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:00.988834 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:00.988851 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:00.988870 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:00.988915 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:00.992276 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.993172 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.993181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.993183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.993343 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:00.993834 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:00.993838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:00.993840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:00.994262 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:00.994499 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:00.994990 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 16:44:00.995339 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 16:44:01.002146 729 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:01.000181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.26.101.1:32987)
0928 16:44:01.000527 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.000532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.000666 (+ 134us) server_negotiation.cc:407] Connection header received
0928 16:44:01.000871 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.000876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.000948 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.001036 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.001905 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33988: BlockingRecv error: recv got EOF from 127.0.0.1:33988 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260928 16:44:01.002584 27028 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc)
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:01.004133 27028 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:01.005442 27028 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:44:01.020846 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.006812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.26.101.1:32987)
0928 16:44:01.007042 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.007048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.007281 (+ 233us) server_negotiation.cc:407] Connection header received
0928 16:44:01.007428 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.007431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.007529 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.007614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.008653 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.009370 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.010672 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.011194 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.011306 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.015306 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.015312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.015316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.015367 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.018133 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.018900 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.018905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.018907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.019047 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.019540 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.019543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.019545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.019835 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.020043 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.020349 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.020540 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 16:44:01.025513 729 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:01.023865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.26.101.1:32987)
0928 16:44:01.024098 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.024102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.024198 (+ 96us) server_negotiation.cc:407] Connection header received
0928 16:44:01.024378 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.024382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.024455 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.024532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.025294 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34000: BlockingRecv error: recv got EOF from 127.0.0.1:34000 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260928 16:44:01.025722 27028 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613789516984-27028-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:01.027393 27028 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:01.028715 27028 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:44:01.044400 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.029974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.26.101.1:32987)
0928 16:44:01.030207 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.030213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.030362 (+ 149us) server_negotiation.cc:407] Connection header received
0928 16:44:01.030502 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.030505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.030572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.030656 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.031789 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.032567 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.033892 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.034359 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.034470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.038365 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.038371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.038374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.038419 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.041503 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.042313 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.042318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.042320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.042507 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.042887 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.042891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.042893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.043299 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.043558 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.043808 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.044035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:44:01.046067 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 319
I20260928 16:44:01.058995 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.046903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.26.101.1:32987)
0928 16:44:01.047106 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.047112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.047439 (+ 327us) server_negotiation.cc:407] Connection header received
0928 16:44:01.047566 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.047639 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.047812 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.047907 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.049014 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.049776 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.051119 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.051616 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.051729 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.053997 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.054002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.054005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.054042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.056700 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.057341 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.057345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.057347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.057538 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.057851 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.057855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.057857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.058120 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.058318 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.058562 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.058733 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:01.059669 445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:01.144894 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 319
I20260928 16:44:01.160423 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 456
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:44:01.176473 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.161339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.26.101.2:44543)
0928 16:44:01.161873 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.161881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.161905 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:44:01.161968 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.161974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.162188 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.162332 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.163453 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.164541 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.165849 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.166427 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.166567 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.170592 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.170607 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.170623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.170664 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.173501 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.174265 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.174273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.174276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.174409 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.174810 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.174814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.174816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.175261 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.175582 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.175857 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.176129 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":246,"threads_started":1}
I20260928 16:44:01.178087 581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:01.261104 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 456
I20260928 16:44:01.276011 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 592
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[32680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:44:01.292949 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:01.277092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.26.101.3:41747)
0928 16:44:01.277834 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:01.277841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:01.277876 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:44:01.277946 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:01.277952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:01.278173 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:01.278412 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:01.279588 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.280654 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.281948 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:01.282569 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:01.282695 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:01.286988 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:01.287004 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:01.287021 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:01.287094 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:01.289889 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.290645 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.290652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.290655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.290786 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:01.291323 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:01.291327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:01.291330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:01.291719 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:01.292026 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:01.292352 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 16:44:01.292642 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":408,"threads_started":1}
I20260928 16:44:01.294262 717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:01.377411 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 592
I20260928 16:44:01.391824 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 32693
I20260928 16:44:01.393121 32756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:01.495951 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 32693
2026-09-28T16:44:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3944 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[749](info): set up 2 sockets
Sep 28 16:44:01 dist-test-slave-t0tw krb5kdc[749](info): commencing operation
krb5kdc: starting...
W20260928 16:44:02.511654 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:02 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab.
Entry for principal oryx/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab.
I20260928 16:44:02.730135 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44153
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab
--principal=oryx/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:03.063999 762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:03.064357 762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:03.064494 762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:03.072238 762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:03.072415 762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:03.072510 762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:03.072588 762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:03.072654 762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:03.083437 762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab
--principal=oryx/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:03.084923 762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:03.086246 762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:03.098201 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:03.098136 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:03.098174 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:03.099252 762 server_base.cc:1034] running on GCE node
I20260928 16:44:03.100147 762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:03.101756 762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:03.102990 762 hybrid_clock.cc:648] HybridClock initialized: now 1790613843102926 us; error 50 us; skew 500 ppm
I20260928 16:44:03.103371 762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:03 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613843, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:03.106662 762 init.cc:377] Logged in from keytab as oryx/127.26.101.62@KRBTEST.COM (short username oryx)
I20260928 16:44:03.108341 762 webserver.cc:460] Webserver started at http://127.26.101.62:38209/ using document root <none> and password file <none>
I20260928 16:44:03.109171 762 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:03.109324 762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:03.109722 762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:03.113021 762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "4c0268f81e1b449e965e51ba616d973d"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "5e0648b5da85a35882c5998e2b117f13"
server_key_iv: "51e819f25cc48f0cb6602c1f17ab6034"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.113813 762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "4c0268f81e1b449e965e51ba616d973d"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "5e0648b5da85a35882c5998e2b117f13"
server_key_iv: "51e819f25cc48f0cb6602c1f17ab6034"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.119290 762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:44:03.123144 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:03.124581 762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:03.124756 762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "4c0268f81e1b449e965e51ba616d973d"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "5e0648b5da85a35882c5998e2b117f13"
server_key_iv: "51e819f25cc48f0cb6602c1f17ab6034"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.124964 762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:03.158437 762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:03.163061 762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:03.163389 762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:03.181246 762 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:44153
I20260928 16:44:03.181269 830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:44153 every 8 connection(s)
I20260928 16:44:03.182675 762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:03.186702 831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:03.192800 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 762
I20260928 16:44:03.193028 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:03.193661 27028 external_mini_cluster.cc:1428] Setting key 742c629ff0af8972a8efb3a4013b5539
I20260928 16:44:03.198154 831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Bootstrap starting.
I20260928 16:44:03.201862 831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:03.203027 831 log.cc:824] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:03.206651 831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: No bootstrap required, opened a new log
Sep 28 16:44:03 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:03.215804 831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } }
I20260928 16:44:03.216303 831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:03.216493 831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0268f81e1b449e965e51ba616d973d, State: Initialized, Role: FOLLOWER
I20260928 16:44:03.217193 831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } }
I20260928 16:44:03.217432 831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:03.217593 831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:03.217777 831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:03.218345 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:03.195796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.26.101.62:44153)
0928 16:44:03.196612 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:03.196626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:03.196670 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:44:03.198370 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:03.198396 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:03.198808 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:03.199189 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:03.201045 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.202418 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.204160 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.205028 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.205188 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:03.210274 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:03.210284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:03.210301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:03.210344 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:03.213610 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:03.214558 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:03.214566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:03.214573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:03.214759 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:03.215206 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:03.215210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:03.215212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:03.215838 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:03.216087 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:03.216654 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 16:44:03.217094 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":371,"threads_started":1}
I20260928 16:44:03.220613 831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } }
I20260928 16:44:03.221220 831 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0268f81e1b449e965e51ba616d973d; no voters:
I20260928 16:44:03.221904 831 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:03.222018 836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:03.222772 836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 LEADER]: Becoming Leader. State: Replica: 4c0268f81e1b449e965e51ba616d973d, State: Running, Role: LEADER
I20260928 16:44:03.223443 836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } }
I20260928 16:44:03.224247 831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:03.226516 838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0268f81e1b449e965e51ba616d973d. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } }
I20260928 16:44:03.226913 838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:03.226799 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } }
I20260928 16:44:03.227216 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:03.230742 844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:03.235752 844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:03.244271 844 catalog_manager.cc:1347] Generated new cluster ID: 6451f1e89b564bf48cbd372a6dc26337
I20260928 16:44:03.244407 844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:03.258599 844 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:03.260619 844 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:03.271322 844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Generated new TSK 0
I20260928 16:44:03.272502 844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
I20260928 16:44:03.328133 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:03.669898 857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:03.670225 857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:03.670361 857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:03.677832 857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:03.677977 857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:03.678118 857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:03.688549 857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:03.689957 857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:03.691349 857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:03.704625 863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:03.704614 865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:03.704998 862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:03.705924 857 server_base.cc:1034] running on GCE node
I20260928 16:44:03.706688 857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:03.707777 857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:03.709010 857 hybrid_clock.cc:648] HybridClock initialized: now 1790613843708958 us; error 47 us; skew 500 ppm
I20260928 16:44:03.709344 857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:03 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613843, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:03.712772 857 init.cc:377] Logged in from keytab as oryx/127.26.101.1@KRBTEST.COM (short username oryx)
I20260928 16:44:03.714327 857 webserver.cc:460] Webserver started at http://127.26.101.1:41593/ using document root <none> and password file <none>
I20260928 16:44:03.715049 857 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:03.715214 857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:03.715534 857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:03.718554 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "8301159e2d577c32b2870b6e63d9cb63"
server_key_iv: "97b658e9cec1e1bffce274525f04c100"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.719408 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "8301159e2d577c32b2870b6e63d9cb63"
server_key_iv: "97b658e9cec1e1bffce274525f04c100"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.725214 857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:03.729218 872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:03.730855 857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:03.731029 857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "8301159e2d577c32b2870b6e63d9cb63"
server_key_iv: "97b658e9cec1e1bffce274525f04c100"
server_key_version: "encryptionkey@0"
I20260928 16:44:03.731279 857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:03.748421 857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:03.753027 857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:03.753311 857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:03.754309 857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:03.755828 857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:03.755904 857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:03.756052 857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:03.756102 857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:03.786419 857 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:44923
I20260928 16:44:03.786458 985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:44923 every 8 connection(s)
I20260928 16:44:03.787856 857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:03.792256 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 857
I20260928 16:44:03.792469 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:03.793121 27028 external_mini_cluster.cc:1428] Setting key a92b3fb4077d561898ad214449f3e149
Sep 28 16:44:03 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613843, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:03.809630 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:03.790572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57587 (local address 127.26.101.62:44153)
0928 16:44:03.791066 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:03.791093 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:03.792099 (+ 1006us) server_negotiation.cc:407] Connection header received
0928 16:44:03.794168 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:03.794174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:03.794276 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:03.794378 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:03.796196 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.796935 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.798518 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.798923 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.799044 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:03.802975 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:03.802981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:03.802985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:03.803028 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:03.805900 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:03.806876 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:03.806880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:03.806883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:03.807032 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:03.807636 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:03.807641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:03.807643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:03.807965 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:03.808092 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:03.809157 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0928 16:44:03.809355 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260928 16:44:03.811326 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:03.791139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:57587)
0928 16:44:03.791914 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:03.791970 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:03.793873 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:03.794665 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:03.794675 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:03.795392 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:03.796015 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:03.796034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.797097 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.797101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:03.798359 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:03.798372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:03.799189 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:03.799197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:03.799447 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:03.800135 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:03.800159 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:03.802659 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:03.806078 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:03.806089 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:03.806109 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:03.806743 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:03.807179 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:03.807183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:03.807187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:03.807457 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:03.808143 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:03.808152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:03.808644 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 16:44:03.808985 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 16:44:03.809619 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":293,"threads_started":1}
I20260928 16:44:03.813171 986 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
WARNING: no policy specified for oryx/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:03.813498 986 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:03.814327 986 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.2@KRBTEST.COM" created.
I20260928 16:44:03.816776 796 ts_manager.cc:194] Registered new tserver with Master: db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:03.818871 796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:57587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
I20260928 16:44:03.843061 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:04.176935 993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:04.177263 993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:04.177410 993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:04.184937 993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:04.185082 993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:04.185242 993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:04.195330 993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:04.196715 993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:04.198113 993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:04.211679 998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.211778 1001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.211796 999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:04.212730 993 server_base.cc:1034] running on GCE node
I20260928 16:44:04.213366 993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:04.214344 993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:04.215591 993 hybrid_clock.cc:648] HybridClock initialized: now 1790613844215532 us; error 42 us; skew 500 ppm
I20260928 16:44:04.215938 993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:04 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:04.219221 993 init.cc:377] Logged in from keytab as oryx/127.26.101.2@KRBTEST.COM (short username oryx)
I20260928 16:44:04.220851 993 webserver.cc:460] Webserver started at http://127.26.101.2:41371/ using document root <none> and password file <none>
I20260928 16:44:04.221590 993 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:04.221719 993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:04.222035 993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:04.225170 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "1d6021cedca46f501f6d0b3591de14b2"
server_key_iv: "b65e9947f5a11a1091c70b7d72c602c8"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.226043 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "1d6021cedca46f501f6d0b3591de14b2"
server_key_iv: "b65e9947f5a11a1091c70b7d72c602c8"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.231657 993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:44:04.235617 1008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.237140 993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:04.237320 993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "1d6021cedca46f501f6d0b3591de14b2"
server_key_iv: "b65e9947f5a11a1091c70b7d72c602c8"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.237528 993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:04.297075 993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:04.301493 993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:04.301792 993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:04.302770 993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:04.304193 993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:04.304276 993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.304397 993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:04.304454 993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.335861 993 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:34975
I20260928 16:44:04.335917 1122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:34975 every 8 connection(s)
I20260928 16:44:04.337215 993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:04.344938 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 993
I20260928 16:44:04.345124 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:04.345580 27028 external_mini_cluster.cc:1428] Setting key 374a0be4f68e457a3547211fbbf43e98
Sep 28 16:44:04 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:04.358739 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.339830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58877 (local address 127.26.101.62:44153)
0928 16:44:04.340275 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:04.340280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:04.341199 (+ 919us) server_negotiation.cc:407] Connection header received
0928 16:44:04.343226 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:04.343233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:04.343333 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:04.343443 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:04.345079 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.345883 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.348062 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.348491 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.348603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:04.352157 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:04.352162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:04.352166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:04.352201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:04.355287 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:04.356170 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:04.356174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:04.356177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:04.356329 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:04.356807 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:04.356811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:04.356813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:04.357116 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:04.357258 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:04.358055 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 16:44:04.358316 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260928 16:44:04.359933 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.340278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:58877)
0928 16:44:04.341033 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:04.341079 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:04.342909 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:04.343715 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:04.343724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:04.344276 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:04.344894 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:04.344914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.346040 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.346044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:04.347920 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:04.347936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.348719 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.348726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:04.348969 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:04.349640 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:04.349664 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:04.351906 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:04.355489 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:04.355498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:04.355515 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:04.356018 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:04.356451 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:04.356455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:04.356458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:04.356680 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:04.357250 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:04.357258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:04.357638 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 16:44:04.357923 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 16:44:04.358569 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":266,"threads_started":1}
I20260928 16:44:04.361570 1123 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
I20260928 16:44:04.361902 1123 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:04.362708 1123 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:44:04.364266 796 ts_manager.cc:194] Registered new tserver with Master: 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975)
I20260928 16:44:04.365113 796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:58877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
I20260928 16:44:04.394388 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:04.728888 1129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:04.729202 1129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:04.729336 1129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:04.736967 1129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:04.737089 1129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:04.737241 1129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:04.747556 1129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:04.748863 1129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:04.750183 1129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:04.763592 1134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.763595 1135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:04.763595 1137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:04.764241 1129 server_base.cc:1034] running on GCE node
I20260928 16:44:04.764935 1129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:04.765915 1129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:04.767159 1129 hybrid_clock.cc:648] HybridClock initialized: now 1790613844767102 us; error 34 us; skew 500 ppm
I20260928 16:44:04.767484 1129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:04 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:04.770660 1129 init.cc:377] Logged in from keytab as oryx/127.26.101.3@KRBTEST.COM (short username oryx)
I20260928 16:44:04.772118 1129 webserver.cc:460] Webserver started at http://127.26.101.3:32863/ using document root <none> and password file <none>
I20260928 16:44:04.772799 1129 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:04.772903 1129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:04.773157 1129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:04.776131 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "e29c93f5f5f000b64bb566fcc33fe506"
server_key_iv: "0e41bdda567a7b98fe778f7a748e83eb"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.776943 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "e29c93f5f5f000b64bb566fcc33fe506"
server_key_iv: "0e41bdda567a7b98fe778f7a748e83eb"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.782295 1129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 16:44:04.786024 1144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.787724 1129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:04.787884 1129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "e29c93f5f5f000b64bb566fcc33fe506"
server_key_iv: "0e41bdda567a7b98fe778f7a748e83eb"
server_key_version: "encryptionkey@0"
I20260928 16:44:04.788082 1129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:04.805987 1129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:04.810405 1129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:04.810694 1129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:04.811781 1129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:04.813230 1129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:04.813311 1129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.813436 1129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:04.813489 1129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:04.822966 986 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
I20260928 16:44:04.844851 1129 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:44177
I20260928 16:44:04.844890 1257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:44177 every 8 connection(s)
I20260928 16:44:04.846174 1129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:04.848049 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1129
I20260928 16:44:04.848217 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:04.848807 27028 external_mini_cluster.cc:1428] Setting key c8b6b9dfdfda2a9c619f4cd6e915cf2c
Sep 28 16:44:04 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613844, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:04.867507 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.848935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35165 (local address 127.26.101.62:44153)
0928 16:44:04.849118 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:04.849123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:04.850740 (+ 1617us) server_negotiation.cc:407] Connection header received
0928 16:44:04.852789 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:04.852794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:04.852898 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:04.852984 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:04.854588 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.855386 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.856970 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.857365 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.857480 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:04.861313 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:04.861320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:04.861326 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:04.861376 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:04.864361 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:04.865194 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:04.865200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:04.865202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:04.865330 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:04.865763 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:04.865768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:04.865770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:04.866059 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:04.866199 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:04.866916 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 16:44:04.867186 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:44:04.868527 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:04.849571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:35165)
0928 16:44:04.850584 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:04.850631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:04.852510 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:04.853252 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:04.853261 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:04.853806 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:04.854407 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:04.854427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.855542 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.855547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:04.856817 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:04.856830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:04.857602 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:04.857610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:04.857835 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:04.858520 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:04.858545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:04.861030 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:04.864511 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:04.864521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:04.864543 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:04.865075 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:04.865438 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:04.865444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:04.865448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:04.865653 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:04.866184 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:04.866193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:04.866558 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 16:44:04.866813 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 16:44:04.867413 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":374,"threads_started":1}
I20260928 16:44:04.869892 1258 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
I20260928 16:44:04.870200 1258 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:04.871001 1258 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
I20260928 16:44:04.872550 796 ts_manager.cc:194] Registered new tserver with Master: bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:04.873363 796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:35165
I20260928 16:44:04.877249 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.26.101.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.61.keytab.
Entry for principal oryx/127.26.101.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.61.keytab.
I20260928 16:44:04.927011 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:37985
--webserver_interface=127.26.101.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.61.keytab
--principal=oryx/127.26.101.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44153,127.26.101.61:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:05.260985 1264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:05.261318 1264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:05.261422 1264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:05.269083 1264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:05.269232 1264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:05.269279 1264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:05.269335 1264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:05.269378 1264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:05.279681 1264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44153,127.26.101.61:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.61.keytab
--principal=oryx/127.26.101.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.101.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
W20260928 16:44:05.280054 1264 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.26.101.62:44153,127.26.101.61:37985'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 16:44:05.281235 1264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:05.282663 1264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:05.294946 1272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.295029 1270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.295047 1269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:05.295943 1264 server_base.cc:1034] running on GCE node
I20260928 16:44:05.296821 1264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:05.298415 1264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:05.299647 1264 hybrid_clock.cc:648] HybridClock initialized: now 1790613845299563 us; error 64 us; skew 500 ppm
I20260928 16:44:05.300005 1264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:05.310467 1264 init.cc:377] Logged in from keytab as oryx/127.26.101.61@KRBTEST.COM (short username oryx)
I20260928 16:44:05.312306 1264 webserver.cc:460] Webserver started at http://127.26.101.61:41829/ using document root <none> and password file <none>
I20260928 16:44:05.313091 1264 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:05.313211 1264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:05.313539 1264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:05.316838 1264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/instance:
uuid: "9aa63bbe490341a98607724bc311a9c4"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "7980cbe1abbea2d4839e985a7bcc6cbb"
server_key_iv: "746da04a3900c2fddf53ce7b19d2d6d9"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.317701 1264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal/instance:
uuid: "9aa63bbe490341a98607724bc311a9c4"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "7980cbe1abbea2d4839e985a7bcc6cbb"
server_key_iv: "746da04a3900c2fddf53ce7b19d2d6d9"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.323575 1264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 16:44:05.327704 1279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:05.329613 1264 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:05.329826 1264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "9aa63bbe490341a98607724bc311a9c4"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "7980cbe1abbea2d4839e985a7bcc6cbb"
server_key_iv: "746da04a3900c2fddf53ce7b19d2d6d9"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.330068 1264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:05.345296 1264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:05.350106 1264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:05.350404 1264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:05.368185 1123 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
I20260928 16:44:05.368883 1264 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.61:37985
I20260928 16:44:05.368925 1331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.61:37985 every 8 connection(s)
I20260928 16:44:05.370718 1264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:05.374751 1332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:05.378726 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1264
I20260928 16:44:05.383344 1332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.61@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:05.401651 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.386661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.26.101.62:44153)
0928 16:44:05.387160 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.387166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.387611 (+ 445us) server_negotiation.cc:407] Connection header received
0928 16:44:05.387796 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.387800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.387869 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.387970 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.389124 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.389883 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.391371 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.392009 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.392162 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.395970 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.395975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.395978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.396015 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.398574 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.399362 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.399367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.399370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.399530 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.399985 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.399989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.399991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.400348 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.400528 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.401145 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.401372 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260928 16:44:05.402915 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.386700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50866)
0928 16:44:05.387470 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.387506 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.387665 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.388049 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.388056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.388725 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.388970 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.388979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.390015 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.390020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.391213 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.391229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.392230 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.392236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.392361 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.392928 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.392953 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.395720 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.398712 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.398721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.398731 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.399246 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.399638 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.399641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.399643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.399819 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.400514 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.400523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.400742 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.400996 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.401669 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":236,"thread_start_us":397,"threads_started":1}
I20260928 16:44:05.402933 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.381289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.26.101.61:37985)
0928 16:44:05.382259 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.382273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.382321 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:44:05.384117 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.384143 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.384600 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.385016 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.386625 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.388392 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.389672 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.390371 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.390540 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.395280 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.395292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.395308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.395353 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.398323 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.399114 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.399124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.399128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.399321 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.399752 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.399757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.399759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.400321 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.400636 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.400989 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.401474 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":657,"thread_start_us":431,"threads_started":1}
I20260928 16:44:05.405082 1332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.61@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
I20260928 16:44:05.418161 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.405766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44882)
0928 16:44:05.405956 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.405971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.406070 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.406381 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.406386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.406792 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.406934 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.406942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.408197 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.408202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.409089 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.409101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.409801 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.409807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.409998 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.410612 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.410629 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.412780 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.415808 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.415812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.415815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.416197 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.416568 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.416572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.416575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.416709 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.417283 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.417287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.417466 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.417596 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.417837 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:44:05.418255 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.405780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.26.101.61:37985)
0928 16:44:05.405951 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.405956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.406015 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:44:05.406177 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.406181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.406249 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.406333 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.407062 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.407794 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.409251 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.409659 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.409771 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.413003 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.413008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.413011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.413043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.415649 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.416301 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.416305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.416308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.416461 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.416835 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.416840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.416842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.417143 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.417268 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.417718 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.417974 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":50}
I20260928 16:44:05.423166 1332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Bootstrap starting.
I20260928 16:44:05.426568 1332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:05.427717 1332 log.cc:824] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:05.430779 1332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: No bootstrap required, opened a new log
I20260928 16:44:05.438750 1332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } }
I20260928 16:44:05.439149 1332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:05.439270 1332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa63bbe490341a98607724bc311a9c4, State: Initialized, Role: FOLLOWER
I20260928 16:44:05.439914 1332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } }
I20260928 16:44:05.440876 1339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } } }
I20260928 16:44:05.441183 1339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:05.442000 1332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:44:05.450894 1350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:05.451045 1350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:05.454404 1264 master_runner.cc:428] Detected that this master 9aa63bbe490341a98607724bc311a9c4 is joining an existing cluster
I20260928 16:44:05.454519 1264 master_runner.cc:432] Initiating AddMaster RPC to add 127.26.101.61:37985
I20260928 16:44:05.481947 1339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:05.482291 1339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } }
I20260928 16:44:05.483981 1339 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153)
I20260928 16:44:05.485790 806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aa63bbe490341a98607724bc311a9c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0268f81e1b449e965e51ba616d973d" is_pre_election: true
I20260928 16:44:05.487154 1281 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 9aa63bbe490341a98607724bc311a9c4; no voters: 4c0268f81e1b449e965e51ba616d973d
WARNING: no policy specified for oryx/127.26.101.60@KRBTEST.COM; defaulting to no policy
I20260928 16:44:05.487914 1339 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 16:44:05.489038 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.460262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.26.101.62:44153)
0928 16:44:05.460483 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.460490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.470273 (+ 9783us) server_negotiation.cc:407] Connection header received
0928 16:44:05.470472 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.470479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.470592 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.470695 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.472308 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.473419 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.475591 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.476247 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.476356 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.479247 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.479254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.479258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.479297 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.484789 (+ 5492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.485959 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.485968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.485972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.486302 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.486799 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.486805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.486809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.487235 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.487514 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.488251 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.488689 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:44:05.490638 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.459642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50874)
0928 16:44:05.470064 (+ 10422us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.470091 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.470322 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.470905 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.470910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.471662 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.472101 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.472114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.473600 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.473608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.475384 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.475400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.476691 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.476699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.476881 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.477737 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.477756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.479021 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.485277 (+ 6256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.485283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.485288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.485797 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.486460 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.486464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.486468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.486659 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.487611 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.487616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.487848 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.489580 (+ 1732us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.490102 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10207,"mutex_wait_us":33,"spinlock_wait_cycles":28544,"thread_start_us":10930,"threads_started":1}
I20260928 16:44:05.490768 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.459723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44892)
0928 16:44:05.470920 (+ 11197us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.470937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.471178 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.471647 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.471650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.472345 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.472659 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.472672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.474062 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.474067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.475639 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.475652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.476865 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.476870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.477024 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.477623 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.477653 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.478978 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.487738 (+ 8760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.487743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.487747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.488296 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.488728 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.488732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.488736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.488908 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.489579 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.489584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.489824 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.489970 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.490282 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11051,"mutex_wait_us":105,"thread_start_us":9941,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.60@KRBTEST.COM" created.
I20260928 16:44:05.490835 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.460290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.26.101.61:37985)
0928 16:44:05.460637 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.460643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.471088 (+ 10445us) server_negotiation.cc:407] Connection header received
0928 16:44:05.471324 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.471329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.471395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.471461 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.472807 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.473822 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.475762 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.476477 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.476566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.479194 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.479201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.479210 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.479246 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.483141 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.488439 (+ 5298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.488445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.488449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.488590 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.489055 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.489059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.489062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.489475 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.489662 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.490085 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.490446 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":235}
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613843, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.62@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
I20260928 16:44:05.517123 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.498807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.26.101.61:37985)
0928 16:44:05.499661 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.499667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.499686 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:05.499761 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.499765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.499847 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.499941 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.501191 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.502322 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.504215 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.504955 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.505100 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.509513 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.509522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.509527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.509570 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.513369 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.514432 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.514439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.514442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.514610 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.515231 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.515240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.515243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.515680 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.515867 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.516586 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.516846 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727}
I20260928 16:44:05.517192 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.498253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44894)
0928 16:44:05.498979 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.499022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.499275 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.500032 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.500039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.500738 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.500994 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.501005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.502518 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.502523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.504049 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.504067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.505166 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.505172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.505361 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.506148 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.506170 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.509220 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.513565 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.513583 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.513593 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.514245 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.514729 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.514734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.514738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.514920 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.515833 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.515843 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.516082 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.516315 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.516771 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":600,"threads_started":1}
I20260928 16:44:05.519492 796 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.26.101.61:37985: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9aa63bbe490341a98607724bc311a9c4"
member_type: NON_VOTER
last_known_addr {
host: "127.26.101.61"
port: 37985
}
attrs {
promote: true
}
}
dest_uuid: "4c0268f81e1b449e965e51ba616d973d"
cas_config_opid_index: -1
I20260928 16:44:05.520781 796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.60.keytab.
Entry for principal oryx/127.26.101.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.60.keytab.
I20260928 16:44:05.523502 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:43107
--webserver_interface=127.26.101.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.60.keytab
--principal=oryx/127.26.101.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:05.532385 1306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:05.534905 1362 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.535619 1306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 16:44:05.536595 1363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:05.539619 1362 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.26.101.61:37985
I20260928 16:44:05.541299 1264 master.cc:537] Master@127.26.101.61:37985 shutting down...
I20260928 16:44:05.542719 1363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.543061 1363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:05.569155 1306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.594373 1339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0268f81e1b449e965e51ba616d973d. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.594663 1339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:05.594383 1369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.594789 1369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:05.594024 1373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.595148 1373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:05.598121 1264 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 16:44:05.598649 1264 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 16:44:05.598832 1264 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: stopping tablet replica
I20260928 16:44:05.617296 1264 master.cc:552] Master@127.26.101.61:37985 shutdown complete.
I20260928 16:44:05.617410 1264 master_runner.cc:305] Clearing existing system tablet
I20260928 16:44:05.617836 1264 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:05.617935 1264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:05.623275 1264 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 16:44:05.625468 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:05.625840 1264 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:05.626219 1264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:05.626344 1264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "9aa63bbe490341a98607724bc311a9c4"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "7980cbe1abbea2d4839e985a7bcc6cbb"
server_key_iv: "746da04a3900c2fddf53ce7b19d2d6d9"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.631466 1264 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:05.638870 1264 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:44:05.639191 1264 log.cc:1196] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 16:44:05.639770 1264 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Deleting consensus metadata
I20260928 16:44:05.640090 1264 master_runner.cc:315] Copying system tablet from 127.26.101.62:44153
I20260928 16:44:05.643477 1264 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: tablet copy: Beginning tablet copy session from remote peer at address 127.26.101.62:44153
I20260928 16:44:05.660391 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.644405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.26.101.62:44153)
0928 16:44:05.644597 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:05.644603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:05.645159 (+ 556us) server_negotiation.cc:407] Connection header received
0928 16:44:05.645397 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:05.645404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:05.645497 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:05.645606 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:05.646683 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.647868 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.649764 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.650479 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.650613 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.653115 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:05.653122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:05.653127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:05.653170 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:05.656886 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.657805 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.657813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.657816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.658020 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:05.658489 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:05.658495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:05.658498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:05.658936 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:05.659136 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:05.659679 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 16:44:05.659992 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:44:05.660948 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:05.644343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50890)
0928 16:44:05.645012 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:05.645035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:05.645213 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:05.645656 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:05.645661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:05.646216 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:05.646499 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.646513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.648036 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.648049 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.649581 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:05.649600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:05.650669 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:05.650677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:05.650874 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:05.651713 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:05.651738 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:05.652868 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:05.657068 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.657075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.657080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.657629 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.658160 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:05.658166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:05.658169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:05.658366 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:05.659155 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:05.659162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:05.659363 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 16:44:05.659535 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:44:05.659930 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":358,"threads_started":1}
I20260928 16:44:05.662153 816 tablet_copy_service.cc:140] P 4c0268f81e1b449e965e51ba616d973d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9aa63bbe490341a98607724bc311a9c4 ({username='oryx', principal='oryx/127.26.101.61@KRBTEST.COM'} at 127.0.0.1:50890)
I20260928 16:44:05.662437 816 tablet_copy_service.cc:161] P 4c0268f81e1b449e965e51ba616d973d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9aa63bbe490341a98607724bc311a9c4 at {username='oryx', principal='oryx/127.26.101.61@KRBTEST.COM'} at 127.0.0.1:50890: session id = 9aa63bbe490341a98607724bc311a9c4-00000000000000000000000000000000
I20260928 16:44:05.666913 816 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:44:05.669557 1264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:44:05.676784 1264 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: tablet copy: Starting download of 0 data blocks...
I20260928 16:44:05.677145 1264 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: tablet copy: Starting download of 1 WAL segments...
I20260928 16:44:05.680450 1264 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:44:05.685376 816 tablet_copy_service.cc:342] P 4c0268f81e1b449e965e51ba616d973d: Request end of tablet copy session 9aa63bbe490341a98607724bc311a9c4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.26.101.61@KRBTEST.COM'} at 127.0.0.1:50890
I20260928 16:44:05.685623 816 tablet_copy_service.cc:433] P 4c0268f81e1b449e965e51ba616d973d: ending tablet copy session 9aa63bbe490341a98607724bc311a9c4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9aa63bbe490341a98607724bc311a9c4
I20260928 16:44:05.697585 1264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:05.701047 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.707399 1383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.711805 1385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:05.714546 1264 server_base.cc:1034] running on GCE node
I20260928 16:44:05.715001 1264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:05.715806 1264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:05.717103 1264 hybrid_clock.cc:648] HybridClock initialized: now 1790613845717046 us; error 40 us; skew 500 ppm
I20260928 16:44:05.717419 1264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:05.719853 1264 init.cc:377] Logged in from keytab as oryx/127.26.101.61@KRBTEST.COM (short username oryx)
I20260928 16:44:05.721315 1264 webserver.cc:460] Webserver started at http://127.26.101.61:39965/ using document root <none> and password file <none>
I20260928 16:44:05.721771 1264 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:05.722002 1264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:05.725680 1264 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:05.727748 1392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:05.728263 1264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:05.728426 1264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "9aa63bbe490341a98607724bc311a9c4"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "7980cbe1abbea2d4839e985a7bcc6cbb"
server_key_iv: "746da04a3900c2fddf53ce7b19d2d6d9"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.728641 1264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:05.751761 1264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:05.756480 1264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:05.782902 1444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.61:37985 every 8 connection(s)
I20260928 16:44:05.782967 1264 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.61:37985
I20260928 16:44:05.784256 1264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
I20260928 16:44:05.789314 1445 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:44:05.793803 1445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Bootstrap starting.
I20260928 16:44:05.809726 1445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:44:05.810441 1445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: Bootstrap complete.
I20260928 16:44:05.811736 1445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } }
I20260928 16:44:05.812134 1445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9aa63bbe490341a98607724bc311a9c4, State: Initialized, Role: LEARNER
I20260928 16:44:05.812496 1445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } }
I20260928 16:44:05.813871 1445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:05.813664 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:05.814136 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:05.821913 1459 catalog_manager.cc:1259] Loaded cluster ID: 6451f1e89b564bf48cbd372a6dc26337
I20260928 16:44:05.822036 1459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: loading cluster ID for follower catalog manager: success
I20260928 16:44:05.825839 1459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: acquiring CA information for follower catalog manager: success
I20260928 16:44:05.829648 1459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:05.876459 1258 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
W20260928 16:44:05.933753 1364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:05.934072 1364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:05.934204 1364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:05.942548 1364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:05.942703 1364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:05.942781 1364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:05.942865 1364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:05.942927 1364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:05.954088 1364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.60.keytab
--principal=oryx/127.26.101.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.101.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:05.955612 1364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:05.956980 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:05.969884 1465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.970075 1467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:05.969844 1464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:05.972031 1364 server_base.cc:1034] running on GCE node
I20260928 16:44:05.972924 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:05.974470 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:05.975692 1364 hybrid_clock.cc:648] HybridClock initialized: now 1790613845975639 us; error 35 us; skew 500 ppm
I20260928 16:44:05.976034 1364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:05 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:05.979480 1364 init.cc:377] Logged in from keytab as oryx/127.26.101.60@KRBTEST.COM (short username oryx)
I20260928 16:44:05.981122 1364 webserver.cc:460] Webserver started at http://127.26.101.60:36467/ using document root <none> and password file <none>
I20260928 16:44:05.981814 1364 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:05.981917 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:05.982178 1364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:05.985216 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/instance:
uuid: "e5d8551c10394afebe9d4220246969fb"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "ff9f7424ff51f4cc30ca1fae04e7d27c"
server_key_iv: "b52def30deed49c288a93561d808774a"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.985939 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal/instance:
uuid: "e5d8551c10394afebe9d4220246969fb"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "ff9f7424ff51f4cc30ca1fae04e7d27c"
server_key_iv: "b52def30deed49c288a93561d808774a"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.991496 1364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 16:44:05.995591 1474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:05.997092 1364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:05.997252 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "e5d8551c10394afebe9d4220246969fb"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "ff9f7424ff51f4cc30ca1fae04e7d27c"
server_key_iv: "b52def30deed49c288a93561d808774a"
server_key_version: "encryptionkey@0"
I20260928 16:44:05.997500 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:06.028030 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:06.033058 1364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:06.033366 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:06.055361 1364 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.60:43107
I20260928 16:44:06.055413 1526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.60:43107 every 8 connection(s)
I20260928 16:44:06.056768 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:06.058354 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1364
I20260928 16:44:06.061918 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:06.070029 1527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.60@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:06.082659 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.060697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.26.101.60:43107)
0928 16:44:06.061513 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.061526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.061573 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:44:06.063721 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.063747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.064175 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.064574 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.066588 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.068044 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.069319 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.069998 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.070197 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.075056 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.075064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.075102 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.075150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.078303 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.078984 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.078991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.078999 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.079217 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.079608 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.079613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.079616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.080225 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.080538 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.081013 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.081426 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":361,"threads_started":1}
I20260928 16:44:06.087317 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.072630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50896)
0928 16:44:06.073416 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.073475 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.073634 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.074028 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.074033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.074730 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.075001 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.075014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.075997 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.076001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.077160 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.077172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.077964 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.077971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.078164 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.078707 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.078728 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.081175 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.084727 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.084739 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.084747 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.085192 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.085671 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.085675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.085677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.085825 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.086395 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.086400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.086585 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.086742 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.086991 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":370,"threads_started":1}
I20260928 16:44:06.087323 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.072869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.26.101.62:44153)
0928 16:44:06.073082 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.073088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.073588 (+ 500us) server_negotiation.cc:407] Connection header received
0928 16:44:06.073768 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.073774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.073864 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.073973 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.075183 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.075855 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.077306 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.077775 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.077889 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.081390 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.081397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.081402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.081440 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.084591 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.085311 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.085317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.085320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.085543 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.085991 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.085996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.086000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.086297 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.086419 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.086836 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.087028 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:44:06.088627 1527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.60@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
I20260928 16:44:06.103734 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.089380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44898)
0928 16:44:06.089566 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.089581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.089659 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.090328 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.090331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.090708 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.090884 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.090895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.092571 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.092576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.093617 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.093639 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.094349 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.094354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.094506 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.095182 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.095200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.097778 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.101174 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.101179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.101182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.101641 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.102022 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.102026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.102028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.102168 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.102708 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.102713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.102871 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.103016 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.103378 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:44:06.103824 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.089402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.26.101.61:37985)
0928 16:44:06.090075 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.090080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.090095 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:06.090143 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.090146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.090203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.090285 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.091011 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.092399 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.093780 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.094186 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.094288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.098062 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.098069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.098072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.098112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.101005 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.101761 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.101767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.101769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.101923 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.102259 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.102263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.102265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.102567 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.102728 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.103163 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.103381 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":448,"threads_started":1}
I20260928 16:44:06.105115 1527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613845, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.60@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
I20260928 16:44:06.118520 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.105973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.0.0.1:44758)
0928 16:44:06.106152 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.106167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.106262 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.106542 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.106544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.106876 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.107014 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.107022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.108076 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.108081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.108969 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.108981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.109767 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.109772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.109921 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.110502 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.110521 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.112877 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.116050 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.116054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.116057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.116478 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.116838 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.116841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.116844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.117014 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.117539 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.117543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.117694 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.117806 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.118017 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:44:06.118517 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.105967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.26.101.60:43107)
0928 16:44:06.106151 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.106156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.106250 (+ 94us) server_negotiation.cc:407] Connection header received
0928 16:44:06.106350 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.106355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.106430 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.106520 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.107175 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.107940 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.109108 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.109597 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.109699 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.113069 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.113074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.113078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.113109 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.115891 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.116588 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.116593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.116595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.116738 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.117128 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.117133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.117136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.117434 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.117563 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.117897 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.118158 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":58}
I20260928 16:44:06.123879 1527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Bootstrap starting.
I20260928 16:44:06.127648 1527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:06.128964 1527 log.cc:824] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:06.132229 1527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: No bootstrap required, opened a new log
I20260928 16:44:06.136355 1536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:06.137434 1536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:06.142810 1527 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } }
I20260928 16:44:06.143221 1527 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:06.143368 1527 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d8551c10394afebe9d4220246969fb, State: Initialized, Role: FOLLOWER
I20260928 16:44:06.144140 1527 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } }
I20260928 16:44:06.144945 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.136216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44910)
0928 16:44:06.138208 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.138231 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.138478 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.138933 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.138938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.138970 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:06.139269 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.139283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.140361 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.140367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.142551 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.142570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.143984 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.143990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.144200 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.144273 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.144388 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.144551 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":1716,"threads_started":1}
I20260928 16:44:06.145062 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.136696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.26.101.61:37985)
0928 16:44:06.136933 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.136940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.138322 (+ 1382us) server_negotiation.cc:407] Connection header received
0928 16:44:06.138655 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.138660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.138755 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.138856 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:06.139448 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.140176 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.142730 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.143761 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.143910 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.144114 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.144506 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.144687 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:44:06.145480 1539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } } }
I20260928 16:44:06.145967 1539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:06.146314 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0268f81e1b449e965e51ba616d973d. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: NON_VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: true } } }
I20260928 16:44:06.146591 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:06.146836 1527 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:06.147329 1534 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: attempting to promote NON_VOTER 9aa63bbe490341a98607724bc311a9c4 to VOTER
I20260928 16:44:06.148330 1534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } }
I20260928 16:44:06.150774 1420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 LEARNER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:44:06.151626 1536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 16:44:06.155243 1534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } }
I20260928 16:44:06.159581 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } }
I20260928 16:44:06.160750 1536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } }
I20260928 16:44:06.161029 1536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:06.160904 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:06.163697 1420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } }
W20260928 16:44:06.169019 1551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:44:06.169215 1551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:44:06.170228 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } }
I20260928 16:44:06.170466 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:06.173511 1364 master_runner.cc:428] Detected that this master e5d8551c10394afebe9d4220246969fb is joining an existing cluster
I20260928 16:44:06.173616 1364 master_runner.cc:432] Initiating AddMaster RPC to add 127.26.101.60:43107
I20260928 16:44:06.189667 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 857
I20260928 16:44:06.205566 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.178002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.26.101.62:44153)
0928 16:44:06.178171 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.178176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.183251 (+ 5075us) server_negotiation.cc:407] Connection header received
0928 16:44:06.183828 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.183835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.183926 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.183995 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.185564 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.186775 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.191408 (+ 4633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.192315 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.192413 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.197151 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.197158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.197164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.197208 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.201701 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.203111 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.203117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.203121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.203298 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.203727 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.203732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.203735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.204109 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.204273 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.204940 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.205194 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:44:06.206593 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.178766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50912)
0928 16:44:06.183109 (+ 4343us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.183133 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.183663 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.184231 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.184235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.184929 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.185346 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.185363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.187023 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.187032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.188764 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.188782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.192748 (+ 3966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.192754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.192943 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.193670 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.193693 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.195110 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.201835 (+ 6725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.201841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.201846 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.202929 (+ 1083us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.203435 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.203438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.203441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.203595 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.204342 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.204347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.204585 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.205779 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.206126 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4148,"threads_started":1}
I20260928 16:44:06.212610 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.178230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.26.101.61:37985)
0928 16:44:06.178366 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.178372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.189476 (+ 11104us) server_negotiation.cc:407] Connection header received
0928 16:44:06.189720 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.189727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.189808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.189871 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.195311 (+ 5440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.196423 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.199769 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.200464 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.200565 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.205703 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.205710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.205715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.205760 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.209241 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.210090 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.210096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.210099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.210257 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.210754 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.210760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.210764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.211162 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.211301 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.211923 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.212187 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:44:06.214095 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.178984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44916)
0928 16:44:06.189370 (+ 10386us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.189389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.189587 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.190096 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.190101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.190750 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.191208 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.191225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.196630 (+ 5405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.196635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.199539 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.199557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.201986 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.201994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.202180 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.204252 (+ 2072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.204275 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.205453 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.209417 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.209422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.209427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.209945 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.210409 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.210413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.210416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.210615 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.211387 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.211392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.211619 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.213145 (+ 1526us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.213504 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10243,"thread_start_us":11946,"threads_started":1}
I20260928 16:44:06.221814 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.179360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.0.0.1:44766)
0928 16:44:06.206931 (+ 27571us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.206951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.207125 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.207568 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.207573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.208296 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.208554 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.208567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.210526 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.210534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.211924 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.211938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.213183 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.213188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.213370 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.214120 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.214143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.215395 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.218988 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.218993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.218998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.219583 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.219933 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.219938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.219941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.220060 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.220682 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.220687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.220834 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.221008 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.221300 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27460,"thread_start_us":3902,"threads_started":1}
I20260928 16:44:06.221930 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.178666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.26.101.60:43107)
0928 16:44:06.178854 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.178860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.207012 (+ 28152us) server_negotiation.cc:407] Connection header received
0928 16:44:06.207233 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.207237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.207298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.207368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.208707 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.209959 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.212053 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.212814 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.212919 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.215607 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.215612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.215617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.215649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.218860 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.219700 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.219706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.219710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.219828 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.220185 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.220190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.220192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.220552 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.220706 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.221133 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.221448 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":39}
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613843, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.62@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.1@KRBTEST.COM
I20260928 16:44:06.237914 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.217348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.26.101.60:43107)
0928 16:44:06.222886 (+ 5538us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.222891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.222912 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:06.222999 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.223007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.223187 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.224463 (+ 1276us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.224632 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.225619 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.226963 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.227692 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.227808 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.231540 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.231545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.231549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.231578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.234603 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.235604 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.235610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.235613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.235764 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.236346 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.236351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.236354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.236731 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.236870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.237289 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.237501 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5434,"thread_start_us":18902,"threads_started":1}
I20260928 16:44:06.237980 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.217025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.0.0.1:44770)
0928 16:44:06.217226 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.217247 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.217382 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.223416 (+ 6034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.223422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.223968 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.224185 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.224201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.225754 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.225760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.226759 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.226773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.227859 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.227865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.228032 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.228734 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.228758 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.231343 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.234781 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.234786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.234790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.235437 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.235932 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.235938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.235942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.236132 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.236850 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.236856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.237039 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.237171 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.237679 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:44:06.239629 796 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.26.101.60:43107: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e5d8551c10394afebe9d4220246969fb"
member_type: NON_VOTER
last_known_addr {
host: "127.26.101.60"
port: 43107
}
attrs {
promote: true
}
}
dest_uuid: "4c0268f81e1b449e965e51ba616d973d"
cas_config_opid_index: 6
I20260928 16:44:06.240285 796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } }
I20260928 16:44:06.242556 1420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 16:44:06.242858 1502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:06.243992 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.190821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.26.101.1:44923)
0928 16:44:06.205000 (+ 14179us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.205011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.205051 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:44:06.205131 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.205141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.205461 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.207894 (+ 2433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.208184 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.209895 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.226634 (+ 16739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.227678 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.233693 (+ 6015us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.235742 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.235762 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.235782 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.235845 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.239987 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.240959 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.240969 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.240973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.241137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.241625 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.241629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.241633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.242102 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.243356 (+ 1254us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.243489 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.243693 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13948,"thread_start_us":14954,"threads_started":1}
I20260928 16:44:06.245483 981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:06.246168 1502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 16:44:06.246454 1534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 16:44:06.248142 1534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:06.250070 1535 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e5d8551c10394afebe9d4220246969fb (127.26.101.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.250566 1420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e5d8551c10394afebe9d4220246969fb (127.26.101.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.254627 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.255020 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:06.255141 1535 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.26.101.60:43107
I20260928 16:44:06.255869 1364 master.cc:537] Master@127.26.101.60:43107 shutting down...
I20260928 16:44:06.255686 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.256297 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:06.258215 783 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 16:44:06.259850 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.260128 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
W20260928 16:44:06.261202 783 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d -> Peer e5d8551c10394afebe9d4220246969fb (127.26.101.60:43107): Couldn't send request to peer e5d8551c10394afebe9d4220246969fb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:06.285321 1364 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:06.285792 1364 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:06.286015 1364 tablet_replica.cc:330] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: stopping tablet replica
I20260928 16:44:06.293169 1364 master.cc:552] Master@127.26.101.60:43107 shutdown complete.
I20260928 16:44:06.293265 1364 master_runner.cc:305] Clearing existing system tablet
I20260928 16:44:06.293699 1364 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:06.293816 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:06.296975 1364 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:06.298624 1567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:06.299391 1364 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:06.299672 1364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:44:06.299815 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "e5d8551c10394afebe9d4220246969fb"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "ff9f7424ff51f4cc30ca1fae04e7d27c"
server_key_iv: "b52def30deed49c288a93561d808774a"
server_key_version: "encryptionkey@0"
I20260928 16:44:06.304841 1364 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:06.312217 1364 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:44:06.312408 1364 log.cc:1196] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 16:44:06.312911 1364 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Deleting consensus metadata
I20260928 16:44:06.313151 1364 master_runner.cc:315] Copying system tablet from 127.26.101.62:44153
I20260928 16:44:06.315789 1364 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: tablet copy: Beginning tablet copy session from remote peer at address 127.26.101.62:44153
I20260928 16:44:06.338454 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.316617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:50920)
0928 16:44:06.322430 (+ 5813us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.322460 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.322634 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.323053 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.323058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.323650 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.324101 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.324120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.325672 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.325678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.327475 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.327496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.328569 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.328575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.328994 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.329851 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.329882 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.331177 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.335314 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.335320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.335324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.335816 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.336303 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.336308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.336313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.336497 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.337259 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.337264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.337479 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.337629 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.337980 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5663,"thread_start_us":5522,"threads_started":1}
I20260928 16:44:06.339572 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.316708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.26.101.62:44153)
0928 16:44:06.316997 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.317003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.322595 (+ 5592us) server_negotiation.cc:407] Connection header received
0928 16:44:06.322782 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.322788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.322880 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.322997 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.324342 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.325502 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.327658 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.328380 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.328529 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.331402 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.331409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.331414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.331457 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.335155 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.335980 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.335987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.335991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.336160 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.336641 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.336646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.336650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.337040 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.337192 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.338971 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.339256 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 16:44:06.340330 816 tablet_copy_service.cc:140] P 4c0268f81e1b449e965e51ba616d973d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e5d8551c10394afebe9d4220246969fb ({username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:50920)
I20260928 16:44:06.340523 816 tablet_copy_service.cc:161] P 4c0268f81e1b449e965e51ba616d973d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e5d8551c10394afebe9d4220246969fb at {username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:50920: session id = e5d8551c10394afebe9d4220246969fb-00000000000000000000000000000000
I20260928 16:44:06.344044 816 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:44:06.345971 1364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:44:06.353156 1364 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: tablet copy: Starting download of 0 data blocks...
I20260928 16:44:06.353533 1364 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: tablet copy: Starting download of 1 WAL segments...
I20260928 16:44:06.356266 1364 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:44:06.361704 816 tablet_copy_service.cc:342] P 4c0268f81e1b449e965e51ba616d973d: Request end of tablet copy session e5d8551c10394afebe9d4220246969fb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:50920
I20260928 16:44:06.361888 816 tablet_copy_service.cc:433] P 4c0268f81e1b449e965e51ba616d973d: ending tablet copy session e5d8551c10394afebe9d4220246969fb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e5d8551c10394afebe9d4220246969fb
I20260928 16:44:06.373974 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:44:06.376849 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 857
W20260928 16:44:06.380858 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:06.384409 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:06.385967 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:06.387207 1364 server_base.cc:1034] running on GCE node
I20260928 16:44:06.387616 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:06.391407 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:06.392701 1364 hybrid_clock.cc:648] HybridClock initialized: now 1790613846392618 us; error 73 us; skew 500 ppm
I20260928 16:44:06.392949 1364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:06.395139 1364 init.cc:377] Logged in from keytab as oryx/127.26.101.60@KRBTEST.COM (short username oryx)
I20260928 16:44:06.400312 1364 webserver.cc:460] Webserver started at http://127.26.101.60:33675/ using document root <none> and password file <none>
I20260928 16:44:06.400606 1364 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:06.400677 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:06.400682 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:44923
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:44153
--webserver_port=41593
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:06.403051 1364 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:44:06.404664 1584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:06.405104 1364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 16:44:06.405231 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "e5d8551c10394afebe9d4220246969fb"
format_stamp: "Formatted at 2026-09-28 16:44:05 on dist-test-slave-t0tw"
server_key: "ff9f7424ff51f4cc30ca1fae04e7d27c"
server_key_iv: "b52def30deed49c288a93561d808774a"
server_key_version: "encryptionkey@0"
I20260928 16:44:06.405416 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:06.417587 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:06.422240 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:06.449978 1364 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.60:43107
I20260928 16:44:06.450158 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.60:43107 every 8 connection(s)
I20260928 16:44:06.451052 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
I20260928 16:44:06.456423 1637 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:44:06.460402 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Bootstrap starting.
I20260928 16:44:06.479835 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:44:06.480422 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Bootstrap complete.
I20260928 16:44:06.481563 1637 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } }
I20260928 16:44:06.481930 1637 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5d8551c10394afebe9d4220246969fb, State: Initialized, Role: LEARNER
I20260928 16:44:06.482189 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } }
I20260928 16:44:06.483371 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:06.483060 1640 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.483637 1640 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: LEARNER
I20260928 16:44:06.492131 1651 catalog_manager.cc:1259] Loaded cluster ID: 6451f1e89b564bf48cbd372a6dc26337
I20260928 16:44:06.492211 1651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: loading cluster ID for follower catalog manager: success
I20260928 16:44:06.494675 1651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: acquiring CA information for follower catalog manager: success
I20260928 16:44:06.496436 1651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:44:06.779799 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.773832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.0.0.1:44786)
0928 16:44:06.774278 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.774301 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.774451 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.774901 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.774906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.774936 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:06.775167 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.775180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.776544 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.776550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.777779 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.777795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.778931 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.778938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.779173 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.779207 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.779295 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.779448 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260928 16:44:06.779807 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.774006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.26.101.60:43107)
0928 16:44:06.774588 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.774593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.774609 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:06.774672 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.774676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.774749 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.774854 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:06.775314 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.776395 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.777942 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.778740 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.778868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.779142 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.779338 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.779474 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":310,"threads_started":1}
I20260928 16:44:06.781036 1612 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260928 16:44:06.781793 1640 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0268f81e1b449e965e51ba616d973d. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:06.782101 1640 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: LEARNER
W20260928 16:44:06.784260 1589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:06.784555 1589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:06.784699 1589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:06.793669 1589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:06.793871 1589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:06.794063 1589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:06.807981 1589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:44923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=41593
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:06.809417 1589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:06.810716 1589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:06.825237 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:06.825453 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:06.825493 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:06.826931 1589 server_base.cc:1034] running on GCE node
I20260928 16:44:06.827716 1589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:06.828830 1589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:06.830068 1589 hybrid_clock.cc:648] HybridClock initialized: now 1790613846829975 us; error 63 us; skew 500 ppm
I20260928 16:44:06.830479 1589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:06.834227 1589 init.cc:377] Logged in from keytab as oryx/127.26.101.1@KRBTEST.COM (short username oryx)
I20260928 16:44:06.835860 1589 webserver.cc:460] Webserver started at http://127.26.101.1:41593/ using document root <none> and password file <none>
I20260928 16:44:06.836609 1589 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:06.836751 1589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:06.842612 1589 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:44:06.845853 1669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:06.847407 1589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:44:06.847585 1589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
format_stamp: "Formatted at 2026-09-28 16:44:03 on dist-test-slave-t0tw"
server_key: "8301159e2d577c32b2870b6e63d9cb63"
server_key_iv: "97b658e9cec1e1bffce274525f04c100"
server_key_version: "encryptionkey@0"
I20260928 16:44:06.848629 1589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:06.874522 1589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:06.879099 1589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:06.879419 1589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:06.880702 1589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:06.882419 1589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:06.882578 1589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:06.882706 1589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:06.882808 1589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:06.922003 1589 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:44923
I20260928 16:44:06.922125 1783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:44923 every 8 connection(s)
I20260928 16:44:06.923974 1589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:06.931643 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1589
I20260928 16:44:06.932025 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 993
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.2@KRBTEST.COM
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
Sep 28 16:44:06 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613846, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
I20260928 16:44:06.968400 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.933576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.26.101.2:34975)
0928 16:44:06.941687 (+ 8111us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.941697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.941730 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:44:06.941808 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.941816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.942099 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.942329 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.945216 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.946977 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.949262 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.950372 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.950553 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.958316 (+ 7763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.958338 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.958358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.958409 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.963606 (+ 5197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.964850 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.964861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.964865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.965046 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.965548 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.965554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.965556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.966038 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.966436 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.967691 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.967982 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7865,"thread_start_us":7679,"threads_started":1}
I20260928 16:44:06.968452 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.928747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:43495 (local address 127.26.101.62:44153)
0928 16:44:06.938959 (+ 10212us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.938966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.945088 (+ 6122us) server_negotiation.cc:407] Connection header received
0928 16:44:06.945269 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.945274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.945368 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.945473 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.948012 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.949199 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.952382 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.953048 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.953150 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.959907 (+ 6757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.959915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.959920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.959970 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.963843 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.965404 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.965410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.965412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.965574 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.966161 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.966166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.966169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.966542 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.966674 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.967849 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.968091 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10073,"thread_start_us":9956,"threads_started":1}
I20260928 16:44:06.968536 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.928844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:42993 (local address 127.26.101.61:37985)
0928 16:44:06.929373 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.929379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.943760 (+ 14381us) server_negotiation.cc:407] Connection header received
0928 16:44:06.943847 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.943854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.943948 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.944099 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.948012 (+ 3913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.949199 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.952382 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.953057 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.953150 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.958973 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.958981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.958986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.959031 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.962852 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.965397 (+ 2545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.965403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.965407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.965576 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.966173 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.966179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.966182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.966536 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.966685 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.967888 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.968148 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":1166,"threads_started":1}
I20260928 16:44:06.970095 1118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:06.970865 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.929416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:43495)
0928 16:44:06.944945 (+ 15529us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.944961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.945134 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.945705 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.945708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.946528 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.947762 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.947794 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.949541 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.949546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.952195 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.952211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.953660 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.953668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.953912 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.954941 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.954964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.959617 (+ 4653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.964041 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.964046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.964049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.965255 (+ 1206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.965728 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.965732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.965735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.966008 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.966771 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.966779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.967367 (+ 588us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.968747 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.969253 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15388,"mutex_wait_us":345,"spinlock_wait_cycles":47616,"thread_start_us":14924,"threads_started":1}
I20260928 16:44:06.971714 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.928823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:41397 (local address 127.26.101.60:43107)
0928 16:44:06.929204 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:06.929210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:06.952502 (+ 23292us) server_negotiation.cc:407] Connection header received
0928 16:44:06.952929 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:06.952935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:06.953020 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:06.953095 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:06.955005 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.956092 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.958106 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.958675 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.958755 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.963469 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:06.963476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:06.963481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:06.963527 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:06.967385 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.968212 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.968219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.968223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.968371 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:06.969210 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:06.969216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:06.969219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:06.970179 (+ 960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:06.970332 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:06.970914 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 16:44:06.971225 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260928 16:44:06.972556 1785 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
I20260928 16:44:06.972952 1785 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:06.973932 1785 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
I20260928 16:44:06.975920 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.929416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.26.101.1:42993)
0928 16:44:06.939977 (+ 10561us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.940053 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.943331 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.945554 (+ 2223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.945567 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.946528 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.947762 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.947794 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.949536 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.949542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.952195 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.952211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.953568 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.953575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.953912 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.954891 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.954924 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.958635 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.963095 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.963107 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.963129 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.965241 (+ 2112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.965733 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.965738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.965741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.966016 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.966778 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.966782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.967369 (+ 587us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.968746 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.975254 (+ 6508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10074,"mutex_wait_us":277,"spinlock_wait_cycles":7296,"thread_start_us":9777,"threads_started":1}
I20260928 16:44:06.976267 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:06.929803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.26.101.1:41397)
0928 16:44:06.952369 (+ 22566us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:06.952391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:06.952758 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:06.953458 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:06.953464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:06.954368 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:06.954803 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.954820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.956287 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.956292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.957939 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:06.957953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:06.959135 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:06.959142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:06.959361 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:06.960220 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:06.960245 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:06.963211 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:06.967586 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.967590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.967594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.968073 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.968879 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:06.968885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:06.968889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:06.969068 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:06.970396 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:06.970401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:06.970584 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 16:44:06.971894 (+ 1310us) client_negotiation.cc:240] Negotiation successful
0928 16:44:06.975408 (+ 3514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22296,"mutex_wait_us":3195,"spinlock_wait_cycles":107520,"thread_start_us":26945,"threads_started":1}
I20260928 16:44:06.976313 796 ts_manager.cc:194] Re-registered known tserver with Master: db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:06.977077 1786 heartbeater.cc:347] Connected to a master server at 127.26.101.61:37985
I20260928 16:44:06.977190 1786 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:06.977638 1786 heartbeater.cc:511] Master 127.26.101.61:37985 requested a full tablet report, sending...
I20260928 16:44:06.977632 796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:43495
I20260928 16:44:06.979938 1409 ts_manager.cc:194] Registered new tserver with Master: db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:06.980113 1784 heartbeater.cc:347] Connected to a master server at 127.26.101.60:43107
I20260928 16:44:06.980243 1784 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:06.981122 1784 heartbeater.cc:511] Master 127.26.101.60:43107 requested a full tablet report, sending...
I20260928 16:44:06.982718 1602 ts_manager.cc:194] Registered new tserver with Master: db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:07.068743 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 993
I20260928 16:44:07.083927 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:34975
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:44153
--webserver_port=41371
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:07.191769 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: NON_VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: true } } }
I20260928 16:44:07.192137 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:07.195374 1798 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: attempting to promote NON_VOTER e5d8551c10394afebe9d4220246969fb to VOTER
I20260928 16:44:07.196019 1798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } }
I20260928 16:44:07.197947 1612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 LEARNER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 16:44:07.198086 1420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Refusing update from remote peer 4c0268f81e1b449e965e51ba616d973d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 16:44:07.198602 1799 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 16:44:07.199116 1798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 16:44:07.201861 1798 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e5d8551c10394afebe9d4220246969fb (127.26.101.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.202517 1420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e5d8551c10394afebe9d4220246969fb (127.26.101.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.206231 1798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.206741 1798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:07.207162 1798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.207471 1798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:07.209964 1803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.211169 1803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:07.216218 1802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.216605 1802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:07.217096 1612 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e5d8551c10394afebe9d4220246969fb (127.26.101.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.221870 1640 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c0268f81e1b449e965e51ba616d973d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:07.222139 1640 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: FOLLOWER
W20260928 16:44:07.429929 1797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:07.430233 1797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:07.430326 1797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:07.437731 1797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:07.437868 1797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:07.438021 1797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:07.447892 1797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=41371
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:07.449267 1797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:07.450619 1797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:07.464175 1808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:07.464107 1812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:07.464121 1809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:07.465129 1797 server_base.cc:1034] running on GCE node
I20260928 16:44:07.465776 1797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:07.466806 1797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:07.468016 1797 hybrid_clock.cc:648] HybridClock initialized: now 1790613847467931 us; error 75 us; skew 500 ppm
I20260928 16:44:07.468338 1797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:07 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613847, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:07.471602 1797 init.cc:377] Logged in from keytab as oryx/127.26.101.2@KRBTEST.COM (short username oryx)
I20260928 16:44:07.473250 1797 webserver.cc:460] Webserver started at http://127.26.101.2:41371/ using document root <none> and password file <none>
I20260928 16:44:07.473965 1797 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:07.474102 1797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:07.479737 1797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:07.482638 1819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:07.484263 1797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:07.484484 1797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "1d6021cedca46f501f6d0b3591de14b2"
server_key_iv: "b65e9947f5a11a1091c70b7d72c602c8"
server_key_version: "encryptionkey@0"
I20260928 16:44:07.485493 1797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:07.513674 1797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:07.518600 1797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:07.518898 1797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:07.520043 1797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:07.521508 1797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:07.521610 1797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:07.521724 1797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:07.521795 1797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:07.553701 1797 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:34975
I20260928 16:44:07.553778 1932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:34975 every 8 connection(s)
I20260928 16:44:07.555233 1797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:07.556154 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1797
I20260928 16:44:07.556466 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1129
Sep 28 16:44:07 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.3@KRBTEST.COM
I20260928 16:44:07.595302 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.557909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.26.101.3:44177)
0928 16:44:07.570116 (+ 12207us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.570127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.570164 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:44:07.570254 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.570264 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.570654 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.573691 (+ 3037us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.574012 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.575659 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.578722 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.579821 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.586613 (+ 6792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.587286 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.587307 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.587324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:07.587374 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.591670 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.592657 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.592666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.592670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.592850 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.593316 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.593320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.593323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.593773 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.594080 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.594457 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.594763 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11936,"thread_start_us":12917,"threads_started":1}
I20260928 16:44:07.597913 1253 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 28 16:44:07 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613847, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
Sep 28 16:44:07 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613847, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
Sep 28 16:44:07 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613847, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:07.610409 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.570864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:57909 (local address 127.26.101.61:37985)
0928 16:44:07.584004 (+ 13140us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.584011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.587803 (+ 3792us) server_negotiation.cc:407] Connection header received
0928 16:44:07.588044 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.588050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.588161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.588243 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.592000 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.593028 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.594882 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.595660 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.595768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.600947 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.600957 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.600962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:07.601033 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.605926 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607066 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.607094 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.607098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.607252 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607874 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.607880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.607884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.608298 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.608465 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.609736 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.609989 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12983,"thread_start_us":12866,"threads_started":1}
I20260928 16:44:07.610543 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.558546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:38033 (local address 127.26.101.62:44153)
0928 16:44:07.568406 (+ 9860us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.568422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.584106 (+ 15684us) server_negotiation.cc:407] Connection header received
0928 16:44:07.587362 (+ 3256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.587369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.587486 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.587585 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.590150 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.590956 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.593762 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.594366 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.594499 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.601670 (+ 7171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.601676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.601679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:07.601716 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.605454 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.606757 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.606763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.606767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.606957 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607819 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.607824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.607828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.608237 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.608394 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.609695 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.609947 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9692,"thread_start_us":9555,"threads_started":1}
I20260928 16:44:07.612133 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.559099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:38033)
0928 16:44:07.581153 (+ 22054us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:07.581233 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:07.586981 (+ 5748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:07.587899 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:07.587924 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:07.588940 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:07.589962 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.589984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.591141 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.591146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.593609 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.593627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.595249 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.595258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.595677 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.596600 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:07.596632 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:07.601409 (+ 4777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:07.605691 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.605701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.605723 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.606615 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.607408 (+ 793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.607416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.607706 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.608834 (+ 1128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:07.608839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:07.609111 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 16:44:07.609411 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 16:44:07.610036 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21611,"mutex_wait_us":36,"spinlock_wait_cycles":46208,"thread_start_us":27713,"threads_started":1}
I20260928 16:44:07.612440 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.570287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.26.101.2:57909)
0928 16:44:07.587240 (+ 16953us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:07.587717 (+ 477us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:07.587902 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:07.588460 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:07.588465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:07.589056 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:07.591788 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.591802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.593232 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.593237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.594708 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.594724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.596153 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.596159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.596390 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.597126 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:07.597147 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:07.600621 (+ 3474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:07.606106 (+ 5485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.606112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.606116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.606866 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.607373 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.607382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.607705 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.608561 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:07.608571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:07.609087 (+ 516us) client_negotiation.cc:769] Sending connection context
0928 16:44:07.609412 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 16:44:07.611811 (+ 2399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16822,"mutex_wait_us":2050,"thread_start_us":15768,"threads_started":1}
I20260928 16:44:07.612455 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.558833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:35611 (local address 127.26.101.60:43107)
0928 16:44:07.585810 (+ 26977us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:07.585818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:07.586885 (+ 1067us) server_negotiation.cc:407] Connection header received
0928 16:44:07.587484 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:07.587490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:07.587579 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:07.587682 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:07.590186 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.591364 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.594771 (+ 3407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.595481 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.595616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.601359 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:07.601367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:07.601371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:07.601429 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:07.605645 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607258 (+ 1613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.607264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.607267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.607468 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:07.608632 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:07.608638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:07.608642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:07.609022 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:07.609158 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:07.611781 (+ 2623us) server_negotiation.cc:299] Negotiation successful
0928 16:44:07.612003 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26832,"thread_start_us":26655,"threads_started":1}
I20260928 16:44:07.612759 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:07.559642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.26.101.2:35611)
0928 16:44:07.586448 (+ 26806us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:07.586470 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:07.587183 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:07.587901 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:07.587913 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:07.588947 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:07.589976 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.589992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.591513 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.591518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.593609 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:07.593627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:07.595916 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:07.595924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:07.596123 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:07.596762 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:07.596781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:07.601069 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:07.605788 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.605797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.605800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.607138 (+ 1338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.607607 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:07.607612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:07.607615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:07.608488 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:07.609230 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:07.609239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:07.610746 (+ 1507us) client_negotiation.cc:769] Sending connection context
0928 16:44:07.610968 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:44:07.611299 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26644,"mutex_wait_us":293,"thread_start_us":24393,"threads_started":1}
I20260928 16:44:07.614213 1934 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
I20260928 16:44:07.614279 1935 heartbeater.cc:347] Connected to a master server at 127.26.101.61:37985
I20260928 16:44:07.614326 1933 heartbeater.cc:347] Connected to a master server at 127.26.101.60:43107
I20260928 16:44:07.614609 1934 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:07.614605 1933 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:07.614606 1935 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:07.616029 1935 heartbeater.cc:511] Master 127.26.101.61:37985 requested a full tablet report, sending...
I20260928 16:44:07.616024 1933 heartbeater.cc:511] Master 127.26.101.60:43107 requested a full tablet report, sending...
I20260928 16:44:07.616170 1934 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
I20260928 16:44:07.618002 796 ts_manager.cc:194] Re-registered known tserver with Master: 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975)
I20260928 16:44:07.618352 1602 ts_manager.cc:194] Registered new tserver with Master: 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975)
I20260928 16:44:07.619211 796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:38033
I20260928 16:44:07.619623 1409 ts_manager.cc:194] Registered new tserver with Master: 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975)
I20260928 16:44:07.691526 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1129
I20260928 16:44:07.706655 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:44177
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:44153
--webserver_port=32863
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:07.981935 1785 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
W20260928 16:44:08.060622 1947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:08.060916 1947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:08.061024 1947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:08.068548 1947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:08.068687 1947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:08.068805 1947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:08.078830 1947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:44177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=32863
--tserver_master_addrs=127.26.101.62:44153,127.26.101.61:37985,127.26.101.60:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:08.080233 1947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:08.081594 1947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:08.095477 1952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:08.095808 1955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:08.096174 1953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:08.096408 1947 server_base.cc:1034] running on GCE node
I20260928 16:44:08.097047 1947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:08.098024 1947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:08.099223 1947 hybrid_clock.cc:648] HybridClock initialized: now 1790613848099162 us; error 56 us; skew 500 ppm
I20260928 16:44:08.099571 1947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:08 dist-test-slave-t0tw krb5kdc[749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:08.102782 1947 init.cc:377] Logged in from keytab as oryx/127.26.101.3@KRBTEST.COM (short username oryx)
I20260928 16:44:08.104403 1947 webserver.cc:460] Webserver started at http://127.26.101.3:32863/ using document root <none> and password file <none>
I20260928 16:44:08.105126 1947 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:08.105262 1947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:08.110918 1947 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:08.114035 1962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:08.115851 1947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:44:08.116050 1947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
format_stamp: "Formatted at 2026-09-28 16:44:04 on dist-test-slave-t0tw"
server_key: "e29c93f5f5f000b64bb566fcc33fe506"
server_key_iv: "0e41bdda567a7b98fe778f7a748e83eb"
server_key_version: "encryptionkey@0"
I20260928 16:44:08.117169 1947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:08.147465 1947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:08.152426 1947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:08.152722 1947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:08.153815 1947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:08.155476 1947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:08.155591 1947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:08.155701 1947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:08.155772 1947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:08.193629 1947 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:44177
I20260928 16:44:08.193691 2076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:44177 every 8 connection(s)
I20260928 16:44:08.195102 1947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:08.204036 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 1947
Sep 28 16:44:08 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.60@KRBTEST.COM
Sep 28 16:44:08 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.61@KRBTEST.COM
I20260928 16:44:08.264778 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.229092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.26.101.60:43107)
0928 16:44:08.236287 (+ 7195us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.236294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.238697 (+ 2403us) server_negotiation.cc:407] Connection header received
0928 16:44:08.238785 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.238791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.238885 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.238987 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.246471 (+ 7484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.248196 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.250446 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.251290 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.251408 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.255561 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.255568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.255573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.255616 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.260265 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.261332 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.261339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.261343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.261511 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.262038 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.262045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.262048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.262444 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.262683 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.264054 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.264389 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":380,"server-negotiator.queue_time_us":7053,"spinlock_wait_cycles":9472,"thread_start_us":7025,"threads_started":1}
I20260928 16:44:08.269306 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.225527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.26.101.61:37985)
0928 16:44:08.247188 (+ 21661us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.247194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.247213 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:08.247264 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.247269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.247345 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.249918 (+ 2573us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.250284 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.252097 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.254020 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.256196 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.256862 (+ 666us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.260905 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.260911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.260915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.260954 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.264938 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.266288 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.266295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.266299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.266518 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.266969 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.266975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.266978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.267397 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.268593 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.268731 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.268921 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21506,"thread_start_us":21903,"threads_started":1}
Sep 28 16:44:08 dist-test-slave-t0tw krb5kdc[749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613848, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:08.276908 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.198764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35783 (local address 127.26.101.60:43107)
0928 16:44:08.219844 (+ 21080us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.219852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.227005 (+ 7153us) server_negotiation.cc:407] Connection header received
0928 16:44:08.227117 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.227123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.227231 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.227343 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.246075 (+ 18732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.247597 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.251484 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.252713 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.252819 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.268736 (+ 15917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.268743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.268748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.268789 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.272634 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.273634 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.273640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.273643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.273814 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.274521 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.274526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.274529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.274900 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.275027 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.276182 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.276430 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20924,"thread_start_us":20799,"threads_started":1}
I20260928 16:44:08.278208 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.198079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:54887 (local address 127.26.101.61:37985)
0928 16:44:08.198580 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.198586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.229224 (+ 30638us) server_negotiation.cc:407] Connection header received
0928 16:44:08.229707 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.229715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.229848 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.229941 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.234671 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.235927 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.245768 (+ 9841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.246536 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.246673 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.266382 (+ 19709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.266390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.266396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.266464 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.271395 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.273913 (+ 2518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.273918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.273922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.274062 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.274474 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.274479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.274482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.274884 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.275035 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.277298 (+ 2263us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.277550 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260928 16:44:08.278738 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.198873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.26.101.3:54887)
0928 16:44:08.229079 (+ 30206us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.229104 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.229423 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.230666 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.230682 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.231860 (+ 1178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:08.232933 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.232964 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.242809 (+ 9845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.242816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.245504 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.245532 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.258438 (+ 12906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.258444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.258658 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.262341 (+ 3683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:08.262379 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:08.266059 (+ 3680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:08.272649 (+ 6590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.272661 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.272684 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.273447 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.274190 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.274196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.274199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.274351 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.275142 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:08.275151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:08.275692 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.276624 (+ 932us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.277244 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30043,"mutex_wait_us":34,"spinlock_wait_cycles":33920,"thread_start_us":34199,"threads_started":1}
I20260928 16:44:08.278831 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.198812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.26.101.3:35783)
0928 16:44:08.223184 (+ 24372us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.223264 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.226292 (+ 3028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.233501 (+ 7209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.233507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.234024 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:08.234319 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.234330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.247921 (+ 13591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.247927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.249931 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.249950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.254915 (+ 4965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.254925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.255350 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.260747 (+ 5397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:08.260783 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:08.267319 (+ 6536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:08.272846 (+ 5527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.272852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.272856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.273458 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.273968 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.273973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.273977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.274336 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.275176 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:08.275179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:08.275692 (+ 513us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.277896 (+ 2204us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.278242 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23944,"mutex_wait_us":27,"spinlock_wait_cycles":24832,"thread_start_us":51787,"threads_started":1}
I20260928 16:44:08.280853 2078 heartbeater.cc:347] Connected to a master server at 127.26.101.60:43107
I20260928 16:44:08.281239 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.199505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:50057)
0928 16:44:08.251116 (+ 51611us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.251143 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.251288 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.253088 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.253094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.253971 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:08.254452 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.254466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.256151 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.256157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.258141 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.258161 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.260042 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.260048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.264763 (+ 4715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.265865 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:08.265890 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:08.271508 (+ 5618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:08.275926 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.275931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.275934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.276424 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.279202 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:08.279206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:08.279208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:08.279379 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:08.280161 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:08.280166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:08.280404 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.280538 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.280794 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51448,"thread_start_us":28031,"threads_started":1}
I20260928 16:44:08.281279 2078 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:08.281873 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.198794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:50057 (local address 127.26.101.62:44153)
0928 16:44:08.214086 (+ 15292us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.214093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.251242 (+ 37149us) server_negotiation.cc:407] Connection header received
0928 16:44:08.251414 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.251421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.251519 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.251630 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.254635 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.255823 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.258311 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.259026 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.259197 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.271753 (+ 12556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.271760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.271765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.271810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.275613 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.276562 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.276568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.276572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.276735 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.279531 (+ 2796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.279536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.279540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.279991 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.280149 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.281025 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.281271 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15131,"thread_start_us":15313,"threads_started":1}
I20260928 16:44:08.282250 2078 heartbeater.cc:511] Master 127.26.101.60:43107 requested a full tablet report, sending...
I20260928 16:44:08.282965 2079 heartbeater.cc:347] Connected to a master server at 127.26.101.62:44153
I20260928 16:44:08.283195 2079 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:08.283219 2080 heartbeater.cc:347] Connected to a master server at 127.26.101.61:37985
I20260928 16:44:08.283394 2080 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:08.283512 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.236649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.26.101.62:44153)
0928 16:44:08.259487 (+ 22838us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.259491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.259507 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:08.259551 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.259555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.259616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.259712 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.263219 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.264494 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.267533 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.268250 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.268401 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.273423 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.273431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.273435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.273477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.277640 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.278838 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.278843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.278846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.279016 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.280838 (+ 1822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.280843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.280846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.281187 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.281425 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.282794 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.283038 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22702,"thread_start_us":22590,"threads_started":1}
I20260928 16:44:08.283850 2079 heartbeater.cc:511] Master 127.26.101.62:44153 requested a full tablet report, sending...
I20260928 16:44:08.283965 2080 heartbeater.cc:511] Master 127.26.101.61:37985 requested a full tablet report, sending...
I20260928 16:44:08.284958 795 ts_manager.cc:194] Re-registered known tserver with Master: bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:08.285162 1602 ts_manager.cc:194] Registered new tserver with Master: bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:08.285656 1409 ts_manager.cc:194] Registered new tserver with Master: bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:08.286149 795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:50057
I20260928 16:44:08.288676 795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:08.296268 795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:08.330017 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.318855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.26.101.3:44177)
0928 16:44:08.320132 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.320141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.320178 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:44:08.320372 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.320381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.320700 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.321642 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:08.322001 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.323874 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.326716 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.328403 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.328539 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.328777 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.329357 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.329646 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":913,"threads_started":1}
I20260928 16:44:08.331209 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.318572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44177 (local address 127.0.0.1:45418)
0928 16:44:08.319307 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.319326 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.319487 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.320939 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.320944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.320972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:08.321364 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.321380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.324065 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.324072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.326412 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.326441 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.328683 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.328690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.328914 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.328958 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.329066 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.329233 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":325,"threads_started":1}
I20260928 16:44:08.335873 2011 tablet_service.cc:1467] Processing CreateTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 (DEFAULT_TABLE table=test-table [id=791863bada01426594bac87adbab6f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:08.337620 2011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086aa5f6f4446c7b9df5f91f91640b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:08.337992 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.320115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.26.101.2:34975)
0928 16:44:08.328338 (+ 8223us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.328347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.328378 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:44:08.328481 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.328489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.328735 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.329849 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:08.330082 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.331652 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.335219 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.336801 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.336957 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.337191 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.337500 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.337690 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7985,"thread_start_us":7877,"threads_started":1}
I20260928 16:44:08.338297 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.319300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34975 (local address 127.0.0.1:59462)
0928 16:44:08.323333 (+ 4033us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.323365 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.323515 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.329020 (+ 5505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.329025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.329058 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:08.329734 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.329747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.331791 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.331797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.333579 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.333596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.337058 (+ 3462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.337065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.337255 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.337295 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.337421 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.337584 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3833,"thread_start_us":992,"threads_started":1}
I20260928 16:44:08.338382 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.319873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44923 (local address 127.0.0.1:47936)
0928 16:44:08.327596 (+ 7723us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.327617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.327762 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.328913 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.328918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.328946 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:08.331245 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.331263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.333690 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.333695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.335266 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.335283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.337343 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.337348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.337549 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.337584 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.337662 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.337792 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7612,"mutex_wait_us":3287,"thread_start_us":4151,"threads_started":1}
I20260928 16:44:08.339506 2113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.320137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.26.101.1:44923)
0928 16:44:08.325384 (+ 5247us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.325393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.327745 (+ 2352us) server_negotiation.cc:407] Connection header received
0928 16:44:08.327927 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.327937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.328259 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.328564 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:08.331786 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.333491 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.335475 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.336893 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.337020 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.337198 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.338378 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.339124 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5065,"thread_start_us":4910,"threads_started":1}
I20260928 16:44:08.342581 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 (DEFAULT_TABLE table=test-table [id=791863bada01426594bac87adbab6f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:08.342615 1868 tablet_service.cc:1467] Processing CreateTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 (DEFAULT_TABLE table=test-table [id=791863bada01426594bac87adbab6f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:08.343827 1717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086aa5f6f4446c7b9df5f91f91640b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:08.344040 1868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086aa5f6f4446c7b9df5f91f91640b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:08.354740 2115 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Bootstrap starting.
I20260928 16:44:08.357754 2116 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Bootstrap starting.
I20260928 16:44:08.359820 2115 tablet_bootstrap.cc:654] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:08.360589 2117 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Bootstrap starting.
I20260928 16:44:08.361285 2115 log.cc:824] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:08.362216 2116 tablet_bootstrap.cc:654] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:08.363745 2116 log.cc:824] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:08.364328 2115 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: No bootstrap required, opened a new log
I20260928 16:44:08.364646 2115 ts_tablet_manager.cc:1397] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 16:44:08.364919 2117 tablet_bootstrap.cc:654] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:08.366532 2117 log.cc:824] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:08.369598 2117 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: No bootstrap required, opened a new log
I20260928 16:44:08.369871 2117 ts_tablet_manager.cc:1397] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:44:08.370589 2116 tablet_bootstrap.cc:492] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: No bootstrap required, opened a new log
I20260928 16:44:08.370924 2116 ts_tablet_manager.cc:1397] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260928 16:44:08.375599 2115 raft_consensus.cc:348] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.376099 2115 raft_consensus.cc:374] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:08.376199 2115 raft_consensus.cc:729] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc98ed4dc225430080d0d7bd6cf5834e, State: Initialized, Role: FOLLOWER
I20260928 16:44:08.376992 2115 consensus_queue.cc:260] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.378867 2079 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
I20260928 16:44:08.378857 2117 raft_consensus.cc:348] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.379359 2117 raft_consensus.cc:374] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:08.379452 2117 raft_consensus.cc:729] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9832b1fe6ba74332bd2bb7d8a0f92f02, State: Initialized, Role: FOLLOWER
I20260928 16:44:08.380386 2117 consensus_queue.cc:260] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.382651 2116 raft_consensus.cc:348] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.383451 2116 raft_consensus.cc:374] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:08.383567 2116 raft_consensus.cc:729] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9ce73e8c6748848cc5ab364a7ad88f, State: Initialized, Role: FOLLOWER
I20260928 16:44:08.384212 2116 consensus_queue.cc:260] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.385373 2115 ts_tablet_manager.cc:1428] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Time spent starting tablet: real 0.021s user 0.011s sys 0.005s
I20260928 16:44:08.385692 2117 ts_tablet_manager.cc:1428] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260928 16:44:08.385978 1934 heartbeater.cc:503] Master 127.26.101.62:44153 was elected leader, sending a full tablet report...
I20260928 16:44:08.392644 2116 ts_tablet_manager.cc:1428] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Time spent starting tablet: real 0.022s user 0.014s sys 0.008s
W20260928 16:44:08.436693 1788 tablet.cc:2369] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:08.448230 2081 tablet.cc:2369] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:08.496965 2122 raft_consensus.cc:482] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:08.497257 2122 raft_consensus.cc:504] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.499267 2122 leader_election.cc:288] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177), db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:08.509085 2113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.499788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:50197 (local address 127.26.101.1:44923)
0928 16:44:08.500094 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.500100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.500763 (+ 663us) server_negotiation.cc:407] Connection header received
0928 16:44:08.500904 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.500909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.500996 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.501104 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:08.501971 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.503269 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.506631 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.508046 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.508288 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.508361 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.508651 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.508785 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:08.509155 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.499635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:45179 (local address 127.26.101.3:44177)
0928 16:44:08.499857 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.499863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.500674 (+ 811us) server_negotiation.cc:407] Connection header received
0928 16:44:08.500902 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.500908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.501005 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.501104 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:08.501971 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.503232 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.506625 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.508003 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.508246 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.508335 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.508704 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.508833 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:44:08.509390 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.499535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44177 (local address 127.26.101.2:45179)
0928 16:44:08.500501 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.500539 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.500725 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.501129 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.501133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.501171 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:08.501766 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.501783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.503399 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.503405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.506419 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.506443 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.508266 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.508274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.508560 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.508587 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.508660 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.508783 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":279,"spinlock_wait_cycles":896,"thread_start_us":500,"threads_started":1}
I20260928 16:44:08.509608 2125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.499595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44923 (local address 127.26.101.2:50197)
0928 16:44:08.500666 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:08.500682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:08.500784 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:08.501129 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:08.501133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:08.501176 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:08.501768 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.501783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.503399 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.503405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.506419 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:08.506443 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.508267 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.508274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:08.508469 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.508510 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:44:08.508624 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 16:44:08.508784 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":133,"spinlock_wait_cycles":4352,"thread_start_us":820,"threads_started":1}
I20260928 16:44:08.510488 2031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086aa5f6f4446c7b9df5f91f91640b2" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" is_pre_election: true
I20260928 16:44:08.510679 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086aa5f6f4446c7b9df5f91f91640b2" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" is_pre_election: true
I20260928 16:44:08.511034 2031 raft_consensus.cc:2454] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 0.
I20260928 16:44:08.511170 1737 raft_consensus.cc:2454] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 0.
I20260928 16:44:08.511824 1821 leader_election.cc:302] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9832b1fe6ba74332bd2bb7d8a0f92f02, bc98ed4dc225430080d0d7bd6cf5834e; no voters:
I20260928 16:44:08.512318 2122 raft_consensus.cc:2781] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:08.512526 2122 raft_consensus.cc:482] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:08.512630 2122 raft_consensus.cc:3037] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:08.515102 2122 raft_consensus.cc:504] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.515766 2122 leader_election.cc:288] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 election: Requested vote from peers bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177), db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923)
I20260928 16:44:08.516330 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086aa5f6f4446c7b9df5f91f91640b2" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
I20260928 16:44:08.516359 2031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086aa5f6f4446c7b9df5f91f91640b2" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
I20260928 16:44:08.516546 1737 raft_consensus.cc:3037] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:08.516557 2031 raft_consensus.cc:3037] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:08.519362 2031 raft_consensus.cc:2454] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 1.
I20260928 16:44:08.519362 1737 raft_consensus.cc:2454] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 1.
I20260928 16:44:08.519795 1821 leader_election.cc:302] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9832b1fe6ba74332bd2bb7d8a0f92f02, bc98ed4dc225430080d0d7bd6cf5834e; no voters:
I20260928 16:44:08.520143 2122 raft_consensus.cc:2781] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:08.520639 2122 raft_consensus.cc:686] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 LEADER]: Becoming Leader. State: Replica: 9832b1fe6ba74332bd2bb7d8a0f92f02, State: Running, Role: LEADER
I20260928 16:44:08.521437 2122 consensus_queue.cc:237] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } }
I20260928 16:44:08.527026 795 catalog_manager.cc:5495] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } health_report { overall_health: UNKNOWN } } }
W20260928 16:44:08.577651 1938 tablet.cc:2369] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:08.578356 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.572509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.26.101.2:34975)
0928 16:44:08.572746 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.572752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.572773 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:08.572998 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.573002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.573098 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.573244 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:08.573671 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.574499 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.575954 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.576454 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.576579 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.577133 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:08.577717 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:08.577893 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.577981 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.578068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:44:08.587831 2031 raft_consensus.cc:1261] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Refusing update from remote peer 9832b1fe6ba74332bd2bb7d8a0f92f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:08.587899 1737 raft_consensus.cc:1261] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Refusing update from remote peer 9832b1fe6ba74332bd2bb7d8a0f92f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:08.588879 2126 consensus_queue.cc:1035] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:08.589380 2122 consensus_queue.cc:1035] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:08.602380 2129 mvcc.cc:205] Tried to move back new op lower bound from 7334354323803762688 to 7334354323548033024. Current Snapshot: MvccSnapshot[applied={T|T < 7334354323803762688}]
I20260928 16:44:08.603209 2131 mvcc.cc:205] Tried to move back new op lower bound from 7334354323803762688 to 7334354323548033024. Current Snapshot: MvccSnapshot[applied={T|T < 7334354323803762688}]
I20260928 16:44:08.612802 2130 mvcc.cc:205] Tried to move back new op lower bound from 7334354323803762688 to 7334354323548033024. Current Snapshot: MvccSnapshot[applied={T|T < 7334354323803762688}]
I20260928 16:44:08.623432 796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:08.623775 796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:08.631592 796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c0268f81e1b449e965e51ba616d973d: Sending DeleteTablet for 3 replicas of tablet 2086aa5f6f4446c7b9df5f91f91640b2
I20260928 16:44:08.633181 1868 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:08 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:59462
I20260928 16:44:08.633284 2011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:08 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:45418
I20260928 16:44:08.634114 1717 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086aa5f6f4446c7b9df5f91f91640b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:08 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:47936
I20260928 16:44:08.634472 2140 tablet_replica.cc:330] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: stopping tablet replica
I20260928 16:44:08.634495 2139 tablet_replica.cc:330] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: stopping tablet replica
I20260928 16:44:08.634963 2139 raft_consensus.cc:2229] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:08.634958 2140 raft_consensus.cc:2229] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:08.635607 2140 raft_consensus.cc:2258] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:08.635842 2139 raft_consensus.cc:2258] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:08.638145 2140 ts_tablet_manager.cc:1905] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:08.638151 2139 ts_tablet_manager.cc:1905] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:08.643030 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 762
I20260928 16:44:08.644426 826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:08.648485 2139 ts_tablet_manager.cc:1918] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:08.651989 2141 tablet_replica.cc:330] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: stopping tablet replica
I20260928 16:44:08.652380 2139 log.cc:1196] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/2086aa5f6f4446c7b9df5f91f91640b2
I20260928 16:44:08.652769 2141 raft_consensus.cc:2229] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:08.653343 2141 raft_consensus.cc:2258] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:08.655860 2141 ts_tablet_manager.cc:1905] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:08.656263 2139 ts_tablet_manager.cc:1939] T 2086aa5f6f4446c7b9df5f91f91640b2 P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting consensus metadata
I20260928 16:44:08.661218 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.637475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.26.101.61:37985)
0928 16:44:08.637664 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.637670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.641527 (+ 3857us) server_negotiation.cc:407] Connection header received
0928 16:44:08.641813 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.641819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.641974 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.642057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.644950 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.646187 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.648446 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.649189 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.649299 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.653483 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.653490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.653496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.653540 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.657374 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.658422 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.658428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.658432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.658609 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.659130 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.659137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.659140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.659558 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.659798 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.660578 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.660862 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:44:08.661247 2140 ts_tablet_manager.cc:1918] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:08.661440 2140 log.cc:1196] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/2086aa5f6f4446c7b9df5f91f91640b2
I20260928 16:44:08.661969 2140 ts_tablet_manager.cc:1939] T 2086aa5f6f4446c7b9df5f91f91640b2 P bc98ed4dc225430080d0d7bd6cf5834e: Deleting consensus metadata
I20260928 16:44:08.669863 2141 ts_tablet_manager.cc:1918] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:08.670073 2141 log.cc:1196] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/2086aa5f6f4446c7b9df5f91f91640b2
I20260928 16:44:08.670702 2141 ts_tablet_manager.cc:1939] T 2086aa5f6f4446c7b9df5f91f91640b2 P db9ce73e8c6748848cc5ab364a7ad88f: Deleting consensus metadata
I20260928 16:44:08.679665 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.642190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.26.101.60:43107)
0928 16:44:08.642391 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.642396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.662224 (+ 19828us) server_negotiation.cc:407] Connection header received
0928 16:44:08.662527 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.662533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.662622 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.662692 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.664877 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.665915 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.667721 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.668343 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.668459 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.672507 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.672514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.672520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.672570 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.676284 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.677227 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.677233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.677235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.677426 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.677886 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.677893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.677896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.678304 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.678619 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.678995 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.679310 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:44:08.781122 782 catalog_manager.cc:4841] TS db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923): tablet 2086aa5f6f4446c7b9df5f91f91640b2 (table test-table [id=791863bada01426594bac87adbab6f53]) successfully deleted
I20260928 16:44:08.781101 781 catalog_manager.cc:4841] TS bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177): tablet 2086aa5f6f4446c7b9df5f91f91640b2 (table test-table [id=791863bada01426594bac87adbab6f53]) successfully deleted
I20260928 16:44:08.781559 781 catalog_manager.cc:4841] TS 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975): tablet 2086aa5f6f4446c7b9df5f91f91640b2 (table test-table [id=791863bada01426594bac87adbab6f53]) successfully deleted
I20260928 16:44:08.782027 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 762
I20260928 16:44:08.837291 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.811430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.26.101.61:37985)
0928 16:44:08.811907 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.811914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.819387 (+ 7473us) server_negotiation.cc:407] Connection header received
0928 16:44:08.819970 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.819978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.820084 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.820167 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.822861 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.824178 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.826465 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.827161 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.827276 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.830686 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.830695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.830699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.830746 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.834123 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.835109 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.835115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.835119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.835280 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.835716 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.835721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.835725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.836106 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.836349 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.836746 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.836964 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260928 16:44:08.842859 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:08.811684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.26.101.60:43107)
0928 16:44:08.812049 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:08.812056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:08.826655 (+ 14599us) server_negotiation.cc:407] Connection header received
0928 16:44:08.828000 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:08.828007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:08.828102 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:08.828218 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:08.830590 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.831530 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.832859 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:08.833512 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:08.833643 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:08.836560 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:08.836567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:08.836572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:08.836617 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:08.840091 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.840831 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.840836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.840840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.841000 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:08.841309 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:08.841314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:08.841317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:08.841689 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:08.841948 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:08.842240 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 16:44:08.842491 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260928 16:44:09.611845 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:09.610822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:36831)
0928 16:44:09.611422 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:09.611595 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":335,"threads_started":1}
W20260928 16:44:09.612248 2079 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:44153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:09.617311 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:09.616359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:36955)
0928 16:44:09.616904 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:09.617085 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":293,"threads_started":1}
W20260928 16:44:09.617682 1934 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:44153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:09.621333 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:09.619959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:59467)
0928 16:44:09.621007 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:09.621170 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":803,"threads_started":1}
W20260928 16:44:09.621717 1785 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:44153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.130832 2154 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c0268f81e1b449e965e51ba616d973d)
I20260928 16:44:10.131199 2154 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } }
I20260928 16:44:10.132685 2154 leader_election.cc:288] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153), 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61:37985)
I20260928 16:44:10.132792 2155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c0268f81e1b449e965e51ba616d973d)
I20260928 16:44:10.133018 2155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } }
I20260928 16:44:10.133908 2155 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153), e5d8551c10394afebe9d4220246969fb (127.26.101.60:43107)
I20260928 16:44:10.141031 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.132817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51386)
0928 16:44:10.140524 (+ 7707us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.140746 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7523,"threads_started":1}
W20260928 16:44:10.142040 1586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.145735 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.134017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51388)
0928 16:44:10.145198 (+ 11181us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.145423 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11033,"threads_started":1}
W20260928 16:44:10.147547 1394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.148670 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.147984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51396)
0928 16:44:10.148235 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.148448 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 16:44:10.149068 1586 leader_election.cc:334] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153): Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.150348 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.149719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51406)
0928 16:44:10.149904 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.150124 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 16:44:10.150928 1394 leader_election.cc:334] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153): Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.153913 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.133097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.26.101.61:37985)
0928 16:44:10.146156 (+ 13059us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.146164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.146195 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:44:10.146281 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.146289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.146376 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.146509 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:10.147451 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.148724 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.151500 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.152818 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.152959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.153044 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.153429 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.153650 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12577,"thread_start_us":12895,"threads_started":1}
I20260928 16:44:10.154063 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.132963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44958)
0928 16:44:10.141258 (+ 8295us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.141285 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.141489 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.146796 (+ 5307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.146803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.146845 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:10.147188 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.147202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.149010 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.149015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.151321 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.151334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.153013 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.153018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.153236 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.153279 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.153377 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.153612 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8204,"thread_start_us":13631,"threads_started":1}
I20260928 16:44:10.154825 1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5d8551c10394afebe9d4220246969fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9aa63bbe490341a98607724bc311a9c4" is_pre_election: true
I20260928 16:44:10.155177 1420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d8551c10394afebe9d4220246969fb in term 1.
I20260928 16:44:10.155689 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.134284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:43107 (local address 127.0.0.1:44830)
0928 16:44:10.145965 (+ 11681us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.145985 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.146164 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.148552 (+ 2388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.148557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.148586 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:10.148858 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.148869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.150812 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.150816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.152836 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.152847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.154753 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.154761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.154950 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.154975 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.155061 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.155319 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11571,"mutex_wait_us":1,"thread_start_us":11980,"threads_started":1}
I20260928 16:44:10.155731 1585 leader_election.cc:302] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9aa63bbe490341a98607724bc311a9c4, e5d8551c10394afebe9d4220246969fb; no voters: 4c0268f81e1b449e965e51ba616d973d
I20260928 16:44:10.156186 2154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:44:10.156366 2154 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c0268f81e1b449e965e51ba616d973d)
I20260928 16:44:10.156520 2154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:44:10.156838 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.146957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.26.101.60:43107)
0928 16:44:10.148202 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.148208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.148233 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:44:10.148290 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.148295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.148377 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.149226 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:10.149549 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.150646 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.152999 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.154556 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.154682 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.154794 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.155135 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.155281 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":832,"threads_started":1}
I20260928 16:44:10.157617 1612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9aa63bbe490341a98607724bc311a9c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e5d8551c10394afebe9d4220246969fb" is_pre_election: true
I20260928 16:44:10.159650 2154 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } }
I20260928 16:44:10.160189 1612 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9aa63bbe490341a98607724bc311a9c4 in current term 2: Already voted for candidate e5d8551c10394afebe9d4220246969fb in this term.
I20260928 16:44:10.161037 1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5d8551c10394afebe9d4220246969fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9aa63bbe490341a98607724bc311a9c4"
I20260928 16:44:10.161227 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.160601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51416)
0928 16:44:10.160889 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.161043 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 16:44:10.161271 1420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:44:10.163065 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.162504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51426)
0928 16:44:10.162817 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.162909 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260928 16:44:10.163265 1586 leader_election.cc:334] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153): Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
I20260928 16:44:10.163441 2154 leader_election.cc:288] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 election: Requested vote from peers 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153), 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61:37985)
I20260928 16:44:10.164256 1420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d8551c10394afebe9d4220246969fb in term 2.
I20260928 16:44:10.164846 1585 leader_election.cc:302] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9aa63bbe490341a98607724bc311a9c4, e5d8551c10394afebe9d4220246969fb; no voters: 4c0268f81e1b449e965e51ba616d973d
I20260928 16:44:10.164948 1396 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9aa63bbe490341a98607724bc311a9c4; no voters: 4c0268f81e1b449e965e51ba616d973d, e5d8551c10394afebe9d4220246969fb
I20260928 16:44:10.165149 2154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:44:10.165437 2155 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:44:10.165720 2154 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [term 2 LEADER]: Becoming Leader. State: Replica: e5d8551c10394afebe9d4220246969fb, State: Running, Role: LEADER
I20260928 16:44:10.166173 2154 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } }
I20260928 16:44:10.168284 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5d8551c10394afebe9d4220246969fb. Latest consensus state: current_term: 2 leader_uuid: "e5d8551c10394afebe9d4220246969fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:10.168689 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:10.169312 2166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:10.173309 2166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:10.174276 2166 catalog_manager.cc:1259] Loaded cluster ID: 6451f1e89b564bf48cbd372a6dc26337
I20260928 16:44:10.174352 2166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:10.175585 2166 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:10.176398 2166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Loaded TSK: 0
I20260928 16:44:10.177289 2166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:10.232313 1601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:10.240005 1601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:10.244647 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.244183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51452)
0928 16:44:10.244354 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.244477 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 16:44:10.244791 1420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [term 2 FOLLOWER]: Refusing update from remote peer e5d8551c10394afebe9d4220246969fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 16:44:10.245718 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.245328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51468)
0928 16:44:10.245480 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.245569 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:44:10.245680 2164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 16:44:10.245985 1586 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb -> Peer 4c0268f81e1b449e965e51ba616d973d (127.26.101.62:44153): Couldn't send request to peer 4c0268f81e1b449e965e51ba616d973d. Status: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:10.248587 2155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5d8551c10394afebe9d4220246969fb. Latest consensus state: current_term: 2 leader_uuid: "e5d8551c10394afebe9d4220246969fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:10.248859 2155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.250658 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e5d8551c10394afebe9d4220246969fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:10.250982 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:10.251546 2155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e5d8551c10394afebe9d4220246969fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c0268f81e1b449e965e51ba616d973d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44153 } } peers { permanent_uuid: "9aa63bbe490341a98607724bc311a9c4" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37985 } attrs { promote: false } } peers { permanent_uuid: "e5d8551c10394afebe9d4220246969fb" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 43107 } attrs { promote: false } } }
I20260928 16:44:10.251816 2155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa63bbe490341a98607724bc311a9c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:44:10.266932 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.260026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34975 (local address 127.0.0.1:59482)
0928 16:44:10.260676 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.260694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.260822 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.261182 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.261186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.261210 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:10.261523 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.261536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.262863 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.262868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.264418 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.264432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.266085 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.266092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.266296 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.266329 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.266410 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.266572 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":4}
I20260928 16:44:10.267171 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.259331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.26.101.1:44923)
0928 16:44:10.259820 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.259826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.259845 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:10.259919 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.259924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.260001 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.260142 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:10.260754 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.261841 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.264444 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.265728 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.265879 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.266017 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.266679 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.266840 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":246,"threads_started":1}
I20260928 16:44:10.267516 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.259179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44923 (local address 127.0.0.1:47950)
0928 16:44:10.259369 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.259388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.259536 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.260181 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.260186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.260216 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:10.260540 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.260554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.261994 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.261999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.264248 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.264262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.265902 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.265908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.266077 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.266163 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.266259 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.266409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":59}
I20260928 16:44:10.268057 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.260187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.26.101.2:34975)
0928 16:44:10.260677 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.260683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.260766 (+ 83us) server_negotiation.cc:407] Connection header received
0928 16:44:10.260957 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.260963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.261038 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.261652 (+ 614us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:10.261806 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.262722 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.264644 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.265840 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.267526 (+ 1686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.267627 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.267685 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.267825 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260928 16:44:10.268373 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 7822aee694c34449a2aa8d3284bcd24b (DEFAULT_TABLE table=test-table [id=eaae0be9f29149e59eb3865369b2bc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:10.268842 1717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7822aee694c34449a2aa8d3284bcd24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:10.268766 1868 tablet_service.cc:1467] Processing CreateTablet for tablet 7822aee694c34449a2aa8d3284bcd24b (DEFAULT_TABLE table=test-table [id=eaae0be9f29149e59eb3865369b2bc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:10.268949 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.260302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44177 (local address 127.0.0.1:45434)
0928 16:44:10.260979 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.261003 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:10.261121 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:10.261964 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:10.261968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:10.261993 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:10.262406 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.262418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.264164 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.264170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.265773 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:10.265786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.268160 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.268166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:10.268335 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.268366 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:44:10.268445 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:44:10.268594 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":5,"thread_start_us":932,"threads_started":1}
I20260928 16:44:10.268965 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.261170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.26.101.3:44177)
0928 16:44:10.261616 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.261622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.261640 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:10.261704 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.261709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.261798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.262572 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:10.262753 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.263988 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.266331 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.267633 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.268076 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.268515 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.268567 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.268698 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":258,"threads_started":1}
I20260928 16:44:10.269198 1868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7822aee694c34449a2aa8d3284bcd24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:10.272503 2011 tablet_service.cc:1467] Processing CreateTablet for tablet 7822aee694c34449a2aa8d3284bcd24b (DEFAULT_TABLE table=test-table [id=eaae0be9f29149e59eb3865369b2bc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:10.272979 2011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7822aee694c34449a2aa8d3284bcd24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:10.278226 2177 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Bootstrap starting.
I20260928 16:44:10.279623 2178 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Bootstrap starting.
I20260928 16:44:10.280145 2177 tablet_bootstrap.cc:654] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.280516 2179 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Bootstrap starting.
I20260928 16:44:10.281277 2178 tablet_bootstrap.cc:654] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.282248 2179 tablet_bootstrap.cc:654] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:10.282588 2177 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: No bootstrap required, opened a new log
I20260928 16:44:10.282737 2177 ts_tablet_manager.cc:1397] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:44:10.283622 2178 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: No bootstrap required, opened a new log
I20260928 16:44:10.283854 2178 ts_tablet_manager.cc:1397] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 16:44:10.283710 2177 raft_consensus.cc:348] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.284133 2177 raft_consensus.cc:374] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:10.284238 2177 raft_consensus.cc:729] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9832b1fe6ba74332bd2bb7d8a0f92f02, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.284505 2177 consensus_queue.cc:260] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.285024 2178 raft_consensus.cc:348] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.285274 2178 raft_consensus.cc:374] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:10.285284 2179 tablet_bootstrap.cc:492] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: No bootstrap required, opened a new log
I20260928 16:44:10.285362 2178 raft_consensus.cc:729] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9ce73e8c6748848cc5ab364a7ad88f, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.285386 2177 ts_tablet_manager.cc:1428] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 16:44:10.285430 2179 ts_tablet_manager.cc:1397] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 16:44:10.285640 2178 consensus_queue.cc:260] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.286258 2179 raft_consensus.cc:348] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.286504 2179 raft_consensus.cc:374] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:10.286600 2179 raft_consensus.cc:729] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc98ed4dc225430080d0d7bd6cf5834e, State: Initialized, Role: FOLLOWER
I20260928 16:44:10.286871 2179 consensus_queue.cc:260] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.287693 2179 ts_tablet_manager.cc:1428] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:44:10.291613 2078 heartbeater.cc:503] Master 127.26.101.60:43107 was elected leader, sending a full tablet report...
I20260928 16:44:10.294168 2178 ts_tablet_manager.cc:1428] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Time spent starting tablet: real 0.010s user 0.001s sys 0.006s
I20260928 16:44:10.296425 1933 heartbeater.cc:503] Master 127.26.101.60:43107 was elected leader, sending a full tablet report...
I20260928 16:44:10.297613 1784 heartbeater.cc:503] Master 127.26.101.60:43107 was elected leader, sending a full tablet report...
I20260928 16:44:10.298817 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.286151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:47901)
0928 16:44:10.298386 (+ 12235us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.298553 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12087,"threads_started":1}
I20260928 16:44:10.301956 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.299973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:45929)
0928 16:44:10.301704 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.301804 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1015}
I20260928 16:44:10.302039 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.290420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:35827)
0928 16:44:10.301617 (+ 11197us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.301782 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11044,"thread_start_us":10896,"threads_started":1}
I20260928 16:44:10.303188 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.294786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:56629)
0928 16:44:10.302750 (+ 7964us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.302913 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7817,"thread_start_us":7784,"threads_started":1}
I20260928 16:44:10.304338 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.303964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:54039)
0928 16:44:10.304122 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.304206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:44:10.319413 2183 raft_consensus.cc:482] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:10.319682 2183 raft_consensus.cc:504] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.320508 2183 leader_election.cc:288] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923), bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:10.321171 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7822aee694c34449a2aa8d3284bcd24b" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" is_pre_election: true
I20260928 16:44:10.321164 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7822aee694c34449a2aa8d3284bcd24b" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" is_pre_election: true
I20260928 16:44:10.321473 1737 raft_consensus.cc:2454] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 0.
I20260928 16:44:10.321501 2030 raft_consensus.cc:2454] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 0.
I20260928 16:44:10.322036 1821 leader_election.cc:302] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9832b1fe6ba74332bd2bb7d8a0f92f02, bc98ed4dc225430080d0d7bd6cf5834e, db9ce73e8c6748848cc5ab364a7ad88f; no voters:
I20260928 16:44:10.322367 2183 raft_consensus.cc:2781] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:10.322479 2183 raft_consensus.cc:482] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:10.322546 2183 raft_consensus.cc:3037] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.324738 2183 raft_consensus.cc:504] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.325340 2183 leader_election.cc:288] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 election: Requested vote from peers db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923), bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177)
I20260928 16:44:10.325934 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7822aee694c34449a2aa8d3284bcd24b" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc98ed4dc225430080d0d7bd6cf5834e"
I20260928 16:44:10.326004 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7822aee694c34449a2aa8d3284bcd24b" candidate_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9ce73e8c6748848cc5ab364a7ad88f"
I20260928 16:44:10.326126 2030 raft_consensus.cc:3037] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.326257 1737 raft_consensus.cc:3037] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:10.328294 2030 raft_consensus.cc:2454] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 1.
I20260928 16:44:10.328461 1737 raft_consensus.cc:2454] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832b1fe6ba74332bd2bb7d8a0f92f02 in term 1.
I20260928 16:44:10.328960 1821 leader_election.cc:302] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9832b1fe6ba74332bd2bb7d8a0f92f02, bc98ed4dc225430080d0d7bd6cf5834e; no voters:
I20260928 16:44:10.329367 2183 raft_consensus.cc:2781] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:10.329943 2183 raft_consensus.cc:686] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 LEADER]: Becoming Leader. State: Replica: 9832b1fe6ba74332bd2bb7d8a0f92f02, State: Running, Role: LEADER
I20260928 16:44:10.330343 2183 consensus_queue.cc:237] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } }
I20260928 16:44:10.331420 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.330838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:47405)
0928 16:44:10.331044 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.331174 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:44:10.333709 1601 catalog_manager.cc:5495] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832b1fe6ba74332bd2bb7d8a0f92f02" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:10.362562 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.358009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.26.101.2:34975)
0928 16:44:10.358202 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.358206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.358248 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:44:10.358310 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.358314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.358392 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.358460 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:10.358916 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.359742 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.361110 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.361534 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.361630 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.362010 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:10.362123 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:10.362186 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.362277 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.362342 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:10.365921 2030 raft_consensus.cc:1261] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Refusing update from remote peer 9832b1fe6ba74332bd2bb7d8a0f92f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:10.365959 1737 raft_consensus.cc:1261] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Refusing update from remote peer 9832b1fe6ba74332bd2bb7d8a0f92f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:10.366662 2189 consensus_queue.cc:1035] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc98ed4dc225430080d0d7bd6cf5834e" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:10.367010 2183 consensus_queue.cc:1035] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9ce73e8c6748848cc5ab364a7ad88f" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:10.369567 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.369111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:50967)
0928 16:44:10.369299 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.369396 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 16:44:10.369668 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.369111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:34979)
0928 16:44:10.369397 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.369513 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260928 16:44:10.375056 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.374426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.1:36471)
0928 16:44:10.374772 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.374871 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260928 16:44:10.378019 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.377520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.3:55123)
0928 16:44:10.377697 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.377827 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:44:10.379822 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.379297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:43795)
0928 16:44:10.379539 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.379633 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 16:44:10.386585 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.386019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.26.101.2:40879)
0928 16:44:10.386258 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.386361 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 16:44:10.386758 1601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:10.387045 1601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:10.392989 1601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb: Sending DeleteTablet for 3 replicas of tablet 7822aee694c34449a2aa8d3284bcd24b
I20260928 16:44:10.393971 1717 tablet_service.cc:1514] Processing DeleteTablet for tablet 7822aee694c34449a2aa8d3284bcd24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:10 UTC) from {username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:47950
I20260928 16:44:10.394156 1868 tablet_service.cc:1514] Processing DeleteTablet for tablet 7822aee694c34449a2aa8d3284bcd24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:10 UTC) from {username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:59482
I20260928 16:44:10.394583 2011 tablet_service.cc:1514] Processing DeleteTablet for tablet 7822aee694c34449a2aa8d3284bcd24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:10 UTC) from {username='oryx', principal='oryx/127.26.101.60@KRBTEST.COM'} at 127.0.0.1:45434
I20260928 16:44:10.395740 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1589
I20260928 16:44:10.396224 2203 tablet_replica.cc:330] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: stopping tablet replica
I20260928 16:44:10.396432 2203 raft_consensus.cc:2229] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:10.396744 2203 raft_consensus.cc:2258] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:10.397956 2203 ts_tablet_manager.cc:1905] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:10.402627 2203 ts_tablet_manager.cc:1918] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:10.402778 2203 log.cc:1196] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/7822aee694c34449a2aa8d3284bcd24b
I20260928 16:44:10.403231 2203 ts_tablet_manager.cc:1939] T 7822aee694c34449a2aa8d3284bcd24b P bc98ed4dc225430080d0d7bd6cf5834e: Deleting consensus metadata
I20260928 16:44:10.404533 1586 catalog_manager.cc:4841] TS bc98ed4dc225430080d0d7bd6cf5834e (127.26.101.3:44177): tablet 7822aee694c34449a2aa8d3284bcd24b (table test-table [id=eaae0be9f29149e59eb3865369b2bc2b]) successfully deleted
I20260928 16:44:10.406374 2202 tablet_replica.cc:330] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: stopping tablet replica
I20260928 16:44:10.406596 2202 raft_consensus.cc:2229] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:10.407042 2202 raft_consensus.cc:2258] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:10.407682 2201 tablet_replica.cc:330] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: stopping tablet replica
I20260928 16:44:10.407897 2201 raft_consensus.cc:2229] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:10.408221 2201 raft_consensus.cc:2258] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:10.408255 2202 ts_tablet_manager.cc:1905] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:10.409273 2201 ts_tablet_manager.cc:1905] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:10.413499 2202 ts_tablet_manager.cc:1918] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:10.413657 2202 log.cc:1196] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/7822aee694c34449a2aa8d3284bcd24b
I20260928 16:44:10.414119 2202 ts_tablet_manager.cc:1939] T 7822aee694c34449a2aa8d3284bcd24b P 9832b1fe6ba74332bd2bb7d8a0f92f02: Deleting consensus metadata
I20260928 16:44:10.414270 2201 ts_tablet_manager.cc:1918] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:10.414409 2201 log.cc:1196] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/7822aee694c34449a2aa8d3284bcd24b
I20260928 16:44:10.414834 2201 ts_tablet_manager.cc:1939] T 7822aee694c34449a2aa8d3284bcd24b P db9ce73e8c6748848cc5ab364a7ad88f: Deleting consensus metadata
I20260928 16:44:10.415243 1586 catalog_manager.cc:4841] TS 9832b1fe6ba74332bd2bb7d8a0f92f02 (127.26.101.2:34975): tablet 7822aee694c34449a2aa8d3284bcd24b (table test-table [id=eaae0be9f29149e59eb3865369b2bc2b]) successfully deleted
I20260928 16:44:10.416115 1587 catalog_manager.cc:4841] TS db9ce73e8c6748848cc5ab364a7ad88f (127.26.101.1:44923): tablet 7822aee694c34449a2aa8d3284bcd24b (table test-table [id=eaae0be9f29149e59eb3865369b2bc2b]) successfully deleted
I20260928 16:44:10.422299 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.397593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.26.101.1:44923)
0928 16:44:10.397845 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.397851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.403567 (+ 5716us) server_negotiation.cc:407] Connection header received
0928 16:44:10.403882 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.403889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.403980 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.404068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.405906 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.407015 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.408933 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.409627 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.409764 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.416583 (+ 6819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.416600 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.416618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.416660 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.419527 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.420195 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.420202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.420205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.420341 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.420755 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.420760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.420762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.421190 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.421532 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.421801 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.421987 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:10.423017 1779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:10.517112 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1589
I20260928 16:44:10.533134 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1797
I20260928 16:44:10.548099 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.533969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.26.101.2:34975)
0928 16:44:10.534174 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.534179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.534200 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:10.534257 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.534261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.534354 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.534501 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.536038 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.536757 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.538052 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.538490 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.538615 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.541764 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.541791 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.541813 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.541878 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.545020 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.545842 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.545851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.545853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.546006 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.546455 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.546460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.546462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.546862 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.547190 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.547530 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.547759 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:44:10.548774 1928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:10.671186 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1797
I20260928 16:44:10.691586 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1947
I20260928 16:44:10.706008 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.692466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.26.101.3:44177)
0928 16:44:10.692697 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:10.692703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:10.692725 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:10.692794 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:10.692799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:10.692884 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:10.692971 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:10.694333 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.695153 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.696488 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:10.696931 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:10.697045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:10.699802 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:10.699820 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:10.699845 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:10.699882 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:10.702943 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.703873 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.703884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.703887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.704040 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:10.704415 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:10.704420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:10.704423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:10.704817 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:10.705115 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:10.705434 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:44:10.705666 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:44:10.706720 2071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:10.802709 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1947
I20260928 16:44:10.819213 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1264
I20260928 16:44:10.820583 1440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:10.821959 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.821108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51484)
0928 16:44:10.821608 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.821734 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260928 16:44:10.828301 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.827920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:44153 (local address 127.0.0.1:51488)
0928 16:44:10.828070 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.828167 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:44:10.942610 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1264
I20260928 16:44:10.961885 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 1364
I20260928 16:44:10.962852 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:10.962353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37985 (local address 127.0.0.1:44960)
0928 16:44:10.962566 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:10.962669 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 16:44:10.963049 1632 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 16:44:10.963069 1585 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5d8551c10394afebe9d4220246969fb -> Peer 9aa63bbe490341a98607724bc311a9c4 (127.26.101.61:37985): Couldn't send request to peer 9aa63bbe490341a98607724bc311a9c4. Status: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:44:11.086140 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 1364
2026-09-28T16:44:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9594 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:11 dist-test-slave-t0tw krb5kdc[2215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:11 dist-test-slave-t0tw krb5kdc[2215](info): set up 2 sockets
Sep 28 16:44:11 dist-test-slave-t0tw krb5kdc[2215](info): commencing operation
krb5kdc: starting...
W20260928 16:44:12.103686 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:12 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab.
Entry for principal oryx/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab.
I20260928 16:44:12.330734 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40427
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab
--principal=oryx/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:12.655652 2229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:12.655970 2229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:12.656090 2229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:12.663643 2229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:12.663772 2229 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:44:12.663815 2229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:12.663918 2229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:12.663991 2229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:12.664080 2229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:12.675009 2229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:40427
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.62.keytab
--principal=oryx/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:12.676427 2229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:12.677878 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:12.690474 2234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:12.690423 2237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:12.690379 2235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:12.691823 2229 server_base.cc:1034] running on GCE node
I20260928 16:44:12.692703 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:12.694273 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:12.695544 2229 hybrid_clock.cc:648] HybridClock initialized: now 1790613852695476 us; error 46 us; skew 500 ppm
I20260928 16:44:12.695902 2229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:12 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:12.699182 2229 init.cc:377] Logged in from keytab as oryx/127.26.101.62@KRBTEST.COM (short username oryx)
I20260928 16:44:12.700742 2229 webserver.cc:460] Webserver started at http://127.26.101.62:37729/ using document root <none> and password file <none>
I20260928 16:44:12.701536 2229 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:12.701646 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:12.701982 2229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:12.705181 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "9d319fe489ee407cafb5ea40e3a88fe2"
format_stamp: "Formatted at 2026-09-28 16:44:12 on dist-test-slave-t0tw"
server_key: "bea4215fa04c7bce5f0772c6f43283bf"
server_key_iv: "15d2b31080448863394aaaf7deb52239"
server_key_version: "encryptionkey@0"
I20260928 16:44:12.706005 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "9d319fe489ee407cafb5ea40e3a88fe2"
format_stamp: "Formatted at 2026-09-28 16:44:12 on dist-test-slave-t0tw"
server_key: "bea4215fa04c7bce5f0772c6f43283bf"
server_key_iv: "15d2b31080448863394aaaf7deb52239"
server_key_version: "encryptionkey@0"
I20260928 16:44:12.711977 2229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 16:44:12.715709 2244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:12.717185 2229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 16:44:12.717375 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "9d319fe489ee407cafb5ea40e3a88fe2"
format_stamp: "Formatted at 2026-09-28 16:44:12 on dist-test-slave-t0tw"
server_key: "bea4215fa04c7bce5f0772c6f43283bf"
server_key_iv: "15d2b31080448863394aaaf7deb52239"
server_key_version: "encryptionkey@0"
I20260928 16:44:12.717644 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:12.738045 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:12.742812 2229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:12.743131 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:12.764339 2229 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:40427
I20260928 16:44:12.764361 2305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:40427 every 8 connection(s)
I20260928 16:44:12.765764 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:12.769639 2306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:12.771982 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 2229
I20260928 16:44:12.772200 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:12.772996 27028 external_mini_cluster.cc:1428] Setting key 948e0b758a6651e4752d58ecde18a995
I20260928 16:44:12.782477 2306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Bootstrap starting.
I20260928 16:44:12.785915 2306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:12.787005 2306 log.cc:824] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:44:12 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:12.790202 2306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: No bootstrap required, opened a new log
I20260928 16:44:12.796372 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:12.775367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.26.101.62:40427)
0928 16:44:12.776287 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:12.776304 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:12.776354 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:44:12.778213 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:12.778240 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:12.778667 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:12.779106 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:12.780669 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:12.781998 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:12.783417 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:12.784183 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:12.784380 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:12.788709 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:12.788717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:12.788733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:12.788770 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:12.791927 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:12.792834 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:12.792842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:12.792848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:12.793052 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:12.793512 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:12.793516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:12.793518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:12.794149 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:12.794397 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:12.794877 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 16:44:12.795328 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":409,"threads_started":1}
I20260928 16:44:12.798775 2306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } }
I20260928 16:44:12.799235 2306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:12.799336 2306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d319fe489ee407cafb5ea40e3a88fe2, State: Initialized, Role: FOLLOWER
I20260928 16:44:12.799871 2306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } }
I20260928 16:44:12.800096 2306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:12.800223 2306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:12.800395 2306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:12.802609 2306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } }
I20260928 16:44:12.803189 2306 leader_election.cc:302] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d319fe489ee407cafb5ea40e3a88fe2; no voters:
I20260928 16:44:12.803776 2306 leader_election.cc:288] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:12.803917 2311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:12.804625 2311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d319fe489ee407cafb5ea40e3a88fe2, State: Running, Role: LEADER
I20260928 16:44:12.805176 2311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } }
I20260928 16:44:12.805833 2306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:12.808255 2312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } } }
I20260928 16:44:12.808360 2313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d319fe489ee407cafb5ea40e3a88fe2. Latest consensus state: current_term: 1 leader_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d319fe489ee407cafb5ea40e3a88fe2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40427 } } }
I20260928 16:44:12.808658 2312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:12.808681 2313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:12.812242 2318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:12.817307 2318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:12.826277 2318 catalog_manager.cc:1347] Generated new cluster ID: 5aad591c10a645a9b999bf63b931b403
I20260928 16:44:12.826407 2318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:12.845808 2318 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:12.847846 2318 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:12.859637 2318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Generated new TSK 0
I20260928 16:44:12.860548 2318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
I20260928 16:44:12.918511 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40427
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:13.254525 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:13.254854 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:13.254987 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:13.262609 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:13.262750 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:13.262836 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:13.262953 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:13.273242 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:13.274581 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:13.275995 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:13.289422 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.289456 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.289484 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:13.290253 2332 server_base.cc:1034] running on GCE node
I20260928 16:44:13.290936 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:13.291966 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:13.293210 2332 hybrid_clock.cc:648] HybridClock initialized: now 1790613853293143 us; error 47 us; skew 500 ppm
I20260928 16:44:13.293534 2332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:13 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:13.297082 2332 init.cc:377] Logged in from keytab as oryx/127.26.101.1@KRBTEST.COM (short username oryx)
I20260928 16:44:13.298616 2332 webserver.cc:460] Webserver started at http://127.26.101.1:41395/ using document root <none> and password file <none>
I20260928 16:44:13.299420 2332 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:13.299530 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:13.299872 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:13.303048 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "de762142feef52d9dde266abbee19007"
server_key_iv: "ccbcc1f630962213fc2542d33aac0019"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.303956 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "de762142feef52d9dde266abbee19007"
server_key_iv: "ccbcc1f630962213fc2542d33aac0019"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.309825 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:44:13.314043 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.315716 2332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:13.315905 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "de762142feef52d9dde266abbee19007"
server_key_iv: "ccbcc1f630962213fc2542d33aac0019"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.316160 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:13.333927 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:13.338666 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:13.338956 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:13.342105 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:13.342204 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.342365 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:13.342449 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:13 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:13.368594 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.343347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.1:43097)
0928 16:44:13.344303 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.344364 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.347328 (+ 2964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.348384 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.348402 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.349257 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.350200 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.350227 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.351565 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.351571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.353622 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.353643 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.354621 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.354631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.354989 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.355965 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.355992 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.359177 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.362843 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.362853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.362872 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.363566 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.364066 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.364071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.364073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.364403 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.365132 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.365142 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.365821 (+ 679us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.366250 (+ 429us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.366887 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":350,"threads_started":1}
I20260928 16:44:13.369225 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.342613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:43097 (local address 127.26.101.62:40427)
0928 16:44:13.343396 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.343403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.344566 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 16:44:13.347772 (+ 3206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.347779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.347900 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.348038 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.350409 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.351381 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.353860 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.354312 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.354426 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.359515 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.359524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.359530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.359577 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.362685 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.363753 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.363759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.363763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.363939 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.364537 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.364542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.364546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.364908 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.365062 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.368683 (+ 3621us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.368956 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":530,"threads_started":1}
I20260928 16:44:13.383831 2332 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41611
I20260928 16:44:13.383875 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41611 every 8 connection(s)
I20260928 16:44:13.385227 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:13.387784 2470 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40427
I20260928 16:44:13.388029 2470 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:13.388720 2470 heartbeater.cc:511] Master 127.26.101.62:40427 requested a full tablet report, sending...
I20260928 16:44:13.390484 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.375897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:39239 (local address 127.26.101.62:40427)
0928 16:44:13.376480 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.376487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.376914 (+ 427us) server_negotiation.cc:407] Connection header received
0928 16:44:13.377206 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.377212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.377313 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.377418 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.378897 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.379899 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.381871 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.382300 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.382381 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.384638 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.384643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.384646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.384679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.387776 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.388499 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.388505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.388509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.388646 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.389033 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.389038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.389042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.389318 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.389473 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.389878 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.390100 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260928 16:44:13.390525 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.375959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.1:39239)
0928 16:44:13.376760 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.376787 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.376989 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.377454 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.377459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.378160 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.378677 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.378691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.380111 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.380120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.381653 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.381673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.382665 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.382670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.382791 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.383532 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.383553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.384445 (+ 892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.387909 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.387914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.387919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.388379 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.388757 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.388763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.388767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.388928 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.389446 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.389451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.389626 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.389750 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.390091 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":534,"threads_started":1}
I20260928 16:44:13.390774 2261 ts_manager.cc:194] Registered new tserver with Master: 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611)
I20260928 16:44:13.391850 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 2332
I20260928 16:44:13.392038 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:13.392719 27028 external_mini_cluster.cc:1428] Setting key f45c0b68d4c578f3f7c84c8194cbba2d
I20260928 16:44:13.392831 2261 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:43097
WARNING: no policy specified for oryx/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
I20260928 16:44:13.442224 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40427
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:13.781803 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:13.782167 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:13.782294 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:13.789723 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:13.789861 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:13.789945 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:13.790027 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:13.800606 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:13.801940 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:13.803448 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:13.817020 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.817039 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:13.817126 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:13.817603 2474 server_base.cc:1034] running on GCE node
I20260928 16:44:13.818303 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:13.819314 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:13.820544 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790613853820461 us; error 51 us; skew 500 ppm
I20260928 16:44:13.820886 2474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:13 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:13.824203 2474 init.cc:377] Logged in from keytab as oryx/127.26.101.2@KRBTEST.COM (short username oryx)
I20260928 16:44:13.825724 2474 webserver.cc:460] Webserver started at http://127.26.101.2:38041/ using document root <none> and password file <none>
I20260928 16:44:13.826433 2474 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:13.826529 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:13.826843 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:13.829973 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "a39f6824d86e4ad98c89bae9f7a4d766"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "0318c6072bae1cfe9833d0e23f343688"
server_key_iv: "e8f933958642ec4efa00f46e776c8d93"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.830756 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "a39f6824d86e4ad98c89bae9f7a4d766"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "0318c6072bae1cfe9833d0e23f343688"
server_key_iv: "e8f933958642ec4efa00f46e776c8d93"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.836202 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 16:44:13.839998 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.841506 2474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:13.841673 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "a39f6824d86e4ad98c89bae9f7a4d766"
format_stamp: "Formatted at 2026-09-28 16:44:13 on dist-test-slave-t0tw"
server_key: "0318c6072bae1cfe9833d0e23f343688"
server_key_iv: "e8f933958642ec4efa00f46e776c8d93"
server_key_version: "encryptionkey@0"
I20260928 16:44:13.841895 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:13.864075 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:13.868510 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:13.868781 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:13.871665 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:13.871759 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:13.871924 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:13.872063 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 16:44:13 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613853, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:13.896763 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.872179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:59657 (local address 127.26.101.62:40427)
0928 16:44:13.872407 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.872415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.874039 (+ 1624us) server_negotiation.cc:407] Connection header received
0928 16:44:13.876320 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.876327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.876427 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.876502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.879143 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.880364 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.882902 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.883568 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.883718 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.889155 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.889161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.889165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.889202 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.892538 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.893762 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.893769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.893772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.893953 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.894524 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.894530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.894533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.894853 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.894999 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.896042 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.896358 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:44:13.897707 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.872936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.2:59657)
0928 16:44:13.873849 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.873906 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.875961 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.877125 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.877138 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.877969 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.878853 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.878885 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.880586 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.880592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.882693 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.882715 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.883881 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.883892 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.884315 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.885277 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.885319 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.888831 (+ 3512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.892728 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.892742 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.892772 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.893596 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.894116 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.894122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.894125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.894403 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.895000 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.895010 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.895551 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.895915 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.896434 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":357,"threads_started":1}
I20260928 16:44:13.918689 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.903757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:56677 (local address 127.26.101.62:40427)
0928 16:44:13.903987 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:13.903994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:13.904566 (+ 572us) server_negotiation.cc:407] Connection header received
0928 16:44:13.904714 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:13.904721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:13.904815 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:13.904937 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:13.905991 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.907470 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.908854 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.909582 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.909728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.912362 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:13.912369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:13.912374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:13.912421 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:13.915391 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.916453 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.916458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.916460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.916626 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:13.917118 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:13.917123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:13.917127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:13.917482 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:13.917629 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:13.918095 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 16:44:13.918338 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:44:13.919453 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:13.903831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.2:56677)
0928 16:44:13.904465 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:13.904484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:13.904603 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:13.904997 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:13.905001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:13.905531 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:13.905839 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.905849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.907635 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.907640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.908697 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:13.908712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:13.909803 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:13.909808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:13.910008 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:13.910777 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:13.910801 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:13.912127 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:13.915605 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.915612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.915616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.916254 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.916812 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:13.916817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:13.916820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:13.916992 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:13.917613 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:13.917619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:13.917835 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 16:44:13.917973 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:44:13.918389 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":306,"threads_started":1}
I20260928 16:44:13.927393 2474 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:43737
I20260928 16:44:13.927433 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:43737 every 8 connection(s)
I20260928 16:44:13.928681 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:13.930883 2611 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40427
I20260928 16:44:13.931154 2611 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:13.931789 2611 heartbeater.cc:511] Master 127.26.101.62:40427 requested a full tablet report, sending...
I20260928 16:44:13.933249 2260 ts_manager.cc:194] Registered new tserver with Master: a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737)
I20260928 16:44:13.934163 2260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:59657
I20260928 16:44:13.935247 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 2474
I20260928 16:44:13.935415 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:13.935859 27028 external_mini_cluster.cc:1428] Setting key 2932ec2d018436d4b219fac8151e1ca2
WARNING: no policy specified for oryx/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
I20260928 16:44:13.982960 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40427
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:14.316860 2615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:14.317169 2615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:14.317319 2615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:14.324885 2615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:14.325033 2615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:14.325165 2615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:44:14.325279 2615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:14.335527 2615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.26.101.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:14.336889 2615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:14.338265 2615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:14.351862 2620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:14.351840 2621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:14.351802 2623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:14.352632 2615 server_base.cc:1034] running on GCE node
I20260928 16:44:14.353259 2615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:14.354257 2615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:14.355491 2615 hybrid_clock.cc:648] HybridClock initialized: now 1790613854355434 us; error 37 us; skew 500 ppm
I20260928 16:44:14.355813 2615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:14 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613854, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:14.359267 2615 init.cc:377] Logged in from keytab as oryx/127.26.101.3@KRBTEST.COM (short username oryx)
I20260928 16:44:14.360726 2615 webserver.cc:460] Webserver started at http://127.26.101.3:43689/ using document root <none> and password file <none>
I20260928 16:44:14.361435 2615 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:14.361526 2615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:14.361809 2615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:14.364890 2615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
format_stamp: "Formatted at 2026-09-28 16:44:14 on dist-test-slave-t0tw"
server_key: "89c6a6660817653541f085b110985bd1"
server_key_iv: "1ed7e4fc4494bef3d2a42f69c6091281"
server_key_version: "encryptionkey@0"
I20260928 16:44:14.365694 2615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
format_stamp: "Formatted at 2026-09-28 16:44:14 on dist-test-slave-t0tw"
server_key: "89c6a6660817653541f085b110985bd1"
server_key_iv: "1ed7e4fc4494bef3d2a42f69c6091281"
server_key_version: "encryptionkey@0"
I20260928 16:44:14.371207 2615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:44:14.374982 2630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:14.376438 2615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:14.376605 2615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
format_stamp: "Formatted at 2026-09-28 16:44:14 on dist-test-slave-t0tw"
server_key: "89c6a6660817653541f085b110985bd1"
server_key_iv: "1ed7e4fc4494bef3d2a42f69c6091281"
server_key_version: "encryptionkey@0"
I20260928 16:44:14.376822 2615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:14.396394 2470 heartbeater.cc:503] Master 127.26.101.62:40427 was elected leader, sending a full tablet report...
I20260928 16:44:14.406450 2615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:14.410909 2615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:14.411224 2615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:14.414103 2615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:14.414211 2615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:14.414359 2615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:14.414448 2615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:44:14 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613854, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:14.436609 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:14.414632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:60553 (local address 127.26.101.62:40427)
0928 16:44:14.414823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.414830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.416367 (+ 1537us) server_negotiation.cc:407] Connection header received
0928 16:44:14.418554 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.418560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.418656 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.418748 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.420996 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.421963 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.424027 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.424443 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.424557 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:14.428894 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:14.428902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:14.428908 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:14.428957 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:14.432561 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:14.433704 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:14.433710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:14.433714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:14.433881 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:14.434378 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:14.434382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:14.434384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:14.434699 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:14.434870 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:14.435889 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0928 16:44:14.436193 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:14.437405 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:14.415391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.3:60553)
0928 16:44:14.416205 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:14.416245 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:14.418196 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:14.419109 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:14.419128 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:14.420056 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:14.420808 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:14.420835 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.422133 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.422139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:14.423856 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:14.423889 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.424670 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.424677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:14.425010 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:14.425852 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:14.425885 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:14.428592 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:14.432734 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:14.432748 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:14.432774 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:14.433562 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:14.434000 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:14.434005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:14.434009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:14.434221 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:14.434841 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:14.434850 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:14.435332 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 16:44:14.435686 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 16:44:14.436151 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":353,"threads_started":1}
I20260928 16:44:14.461658 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:14.443333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:40829 (local address 127.26.101.62:40427)
0928 16:44:14.443534 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.443541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.447756 (+ 4215us) server_negotiation.cc:407] Connection header received
0928 16:44:14.447904 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.447911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.448033 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.448167 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.449155 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.450341 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.452081 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.452877 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.453026 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:14.455204 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:14.455212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:14.455218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:14.455264 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:14.458663 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:14.459691 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:14.459697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:14.459699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:14.459862 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:14.460255 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:14.460259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:14.460261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:14.460582 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:14.460700 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:14.461206 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 16:44:14.461409 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:44:14.462013 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:14.443410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.26.101.3:40829)
0928 16:44:14.447601 (+ 4191us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:14.447626 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:14.447782 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:14.448182 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:14.448185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:14.448696 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:14.449001 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:14.449012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.450517 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.450521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:14.451884 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:14.451906 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.453093 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.453098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:14.453243 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:14.454014 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:14.454040 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:14.454946 (+ 906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:14.458823 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:14.458831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:14.458836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:14.459525 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:14.459969 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:14.459974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:14.459978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:14.460142 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:14.460685 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:14.460690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:14.460922 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 16:44:14.461068 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:44:14.461501 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"thread_start_us":3881,"threads_started":1}
I20260928 16:44:14.474066 2615 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:44881
I20260928 16:44:14.474108 2751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:44881 every 8 connection(s)
I20260928 16:44:14.475446 2615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:14.477543 2752 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40427
I20260928 16:44:14.477782 2752 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:14.478458 2752 heartbeater.cc:511] Master 127.26.101.62:40427 requested a full tablet report, sending...
I20260928 16:44:14.479874 2260 ts_manager.cc:194] Registered new tserver with Master: 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:14.480758 2260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:60553
I20260928 16:44:14.484984 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 2615
I20260928 16:44:14.485145 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:14.485574 27028 external_mini_cluster.cc:1428] Setting key a3ec8c4c223d4f1f6bdaaf9b3ab271fb
I20260928 16:44:14.489190 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:44:14 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:44:14 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:14 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:14.937043 2611 heartbeater.cc:503] Master 127.26.101.62:40427 was elected leader, sending a full tablet report...
I20260928 16:44:15.483511 2752 heartbeater.cc:503] Master 127.26.101.62:40427 was elected leader, sending a full tablet report...
W20260928 16:44:15.587586 2760 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.051s user 0.001s sys 0.014s
W20260928 16:44:15.588902 2357 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:14.530275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.26.101.62:40427)
0928 16:44:14.530513 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:14.530520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:14.530750 (+ 230us) server_negotiation.cc:407] Connection header received
0928 16:44:14.530954 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:14.530959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:14.531028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:14.531140 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:14.532571 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.533323 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.534654 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:14.535149 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:14.535265 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.588592 (+1053327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44594: BlockingRecv error: recv got EOF from 127.0.0.1:44594 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 16:44:15 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:15.616113 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.596922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.26.101.62:40427)
0928 16:44:15.597104 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.597108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.597454 (+ 346us) server_negotiation.cc:407] Connection header received
0928 16:44:15.597618 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.597622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.597723 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.597826 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:15.599155 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.599955 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.601440 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.601933 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.602050 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.610404 (+ 8354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:15.610411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:15.610414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:15.610462 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:15.613223 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:15.614028 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:15.614033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:15.614036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:15.614194 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:15.614681 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:15.614686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:15.614688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:15.615022 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:15.615477 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.615616 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.615806 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:44:15.620445 2260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:15.627576 2260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:15.668694 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.641235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41611 (local address 127.0.0.1:58338)
0928 16:44:15.652475 (+ 11240us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:15.652554 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:15.653984 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:15.659965 (+ 5981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:15.659975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:15.660020 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:15.660567 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.660584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.662640 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.662647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.665166 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.665181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.667475 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.667482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.667748 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.667814 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 16:44:15.667980 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 16:44:15.668217 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10831,"mutex_wait_us":5,"thread_start_us":22717,"threads_started":1}
I20260928 16:44:15.669489 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.641363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.26.101.1:41611)
0928 16:44:15.659134 (+ 17771us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.659147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.659182 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:44:15.659305 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.659315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.659649 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.659895 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:15.660796 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.662432 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.665376 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.666914 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.667045 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.667326 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.668993 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.669180 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17558,"thread_start_us":17408,"threads_started":1}
I20260928 16:44:15.671296 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.643954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44881 (local address 127.0.0.1:50668)
0928 16:44:15.664308 (+ 20354us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:15.664332 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:15.664466 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:15.665050 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:15.665055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:15.665091 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:15.665455 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.665465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.666779 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.666785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.668795 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.668815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.670488 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.670496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.670660 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.670696 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:44:15.670794 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:44:15.670935 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20221,"thread_start_us":14126,"threads_started":1}
I20260928 16:44:15.672104 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.641870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.26.101.2:43737)
0928 16:44:15.663934 (+ 22064us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.663944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.663981 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:44:15.664091 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.664099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.664377 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.664658 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:15.665364 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.666895 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.669558 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.670967 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.671143 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.671343 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.671733 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.671870 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21880,"thread_start_us":21740,"threads_started":1}
I20260928 16:44:15.672098 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.641414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43737 (local address 127.0.0.1:43926)
0928 16:44:15.657787 (+ 16373us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:15.657818 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:15.657982 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:15.664649 (+ 6667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:15.664654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:15.664677 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:15.665130 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.665144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.667107 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.667113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.669331 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.669348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.671242 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.671251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.671426 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.671459 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:44:15.671558 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:44:15.671689 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16193,"mutex_wait_us":27,"thread_start_us":10470,"threads_started":1}
I20260928 16:44:15.673919 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.644250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.26.101.3:44881)
0928 16:44:15.663470 (+ 19220us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.663479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.664455 (+ 976us) server_negotiation.cc:407] Connection header received
0928 16:44:15.664614 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.664623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.664856 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.665060 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:15.665623 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.666627 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.669012 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.670249 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.670398 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.670610 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.673396 (+ 2786us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.673617 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19037,"thread_start_us":18892,"threads_started":1}
I20260928 16:44:15.673959 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 4201229ff82f4e499ee903969e29efaa (DEFAULT_TABLE table=test-table [id=c672c8668ef1433b8f60a98887ee9c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.675563 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4201229ff82f4e499ee903969e29efaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.676167 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 4201229ff82f4e499ee903969e29efaa (DEFAULT_TABLE table=test-table [id=c672c8668ef1433b8f60a98887ee9c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.677336 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4201229ff82f4e499ee903969e29efaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.677681 2682 tablet_service.cc:1467] Processing CreateTablet for tablet 4201229ff82f4e499ee903969e29efaa (DEFAULT_TABLE table=test-table [id=c672c8668ef1433b8f60a98887ee9c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.679183 2682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4201229ff82f4e499ee903969e29efaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.690558 2777 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Bootstrap starting.
I20260928 16:44:15.694092 2777 tablet_bootstrap.cc:654] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.695034 2778 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Bootstrap starting.
I20260928 16:44:15.695535 2777 log.cc:824] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:15.696692 2779 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Bootstrap starting.
I20260928 16:44:15.698949 2778 tablet_bootstrap.cc:654] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.700081 2779 tablet_bootstrap.cc:654] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.700373 2778 log.cc:824] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:15.701215 2779 log.cc:824] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:15.701318 2777 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: No bootstrap required, opened a new log
I20260928 16:44:15.701581 2777 ts_tablet_manager.cc:1397] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:44:15.703778 2778 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: No bootstrap required, opened a new log
I20260928 16:44:15.704085 2778 ts_tablet_manager.cc:1397] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 16:44:15.705206 2779 tablet_bootstrap.cc:492] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: No bootstrap required, opened a new log
I20260928 16:44:15.705544 2779 ts_tablet_manager.cc:1397] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 16:44:15.712321 2777 raft_consensus.cc:348] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.712850 2777 raft_consensus.cc:374] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.713014 2777 raft_consensus.cc:729] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.713642 2779 raft_consensus.cc:348] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.713729 2777 consensus_queue.cc:260] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.713987 2779 raft_consensus.cc:374] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.714116 2779 raft_consensus.cc:729] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26649f9e6a894eaeba2bd1dcc7bc2da7, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.714313 2778 raft_consensus.cc:348] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.714811 2778 raft_consensus.cc:374] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.714915 2778 raft_consensus.cc:729] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.715710 2777 ts_tablet_manager.cc:1428] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 16:44:15.715552 2778 consensus_queue.cc:260] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.716135 2779 consensus_queue.cc:260] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.717525 2778 ts_tablet_manager.cc:1428] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260928 16:44:15.720366 2779 ts_tablet_manager.cc:1428] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
W20260928 16:44:15.727854 2753 tablet.cc:2369] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:15.813287 2783 raft_consensus.cc:482] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:15.813653 2783 raft_consensus.cc:504] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.815393 2783 leader_election.cc:288] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:15.822816 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.816259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:49195 (local address 127.26.101.3:44881)
0928 16:44:15.816487 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.816492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.817099 (+ 607us) server_negotiation.cc:407] Connection header received
0928 16:44:15.817262 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.817268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.817347 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.817441 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:15.818032 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.818780 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.820923 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.821847 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.821993 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.822071 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.822430 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.822565 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:44:15.822834 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.816004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:56437 (local address 127.26.101.1:41611)
0928 16:44:15.816221 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.816228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.816932 (+ 704us) server_negotiation.cc:407] Connection header received
0928 16:44:15.817132 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.817138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.817218 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.817292 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:15.817945 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.818689 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.820877 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.821865 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.822042 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.822113 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.822474 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.822600 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:44:15.822993 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.815862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41611 (local address 127.26.101.2:56437)
0928 16:44:15.816776 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:15.816800 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:15.816974 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:15.817345 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:15.817350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:15.817375 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:15.817773 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.817789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.818830 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.818835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.820714 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.820726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.822108 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.822116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.822306 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.822340 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:44:15.822430 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:44:15.822568 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":89,"thread_start_us":792,"threads_started":1}
I20260928 16:44:15.823032 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.815881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44881 (local address 127.26.101.2:49195)
0928 16:44:15.816968 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:15.816990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:15.817129 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:15.817473 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:15.817477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:15.817511 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:15.817872 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.817886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.818920 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.818924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.820697 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:15.820710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.822062 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.822070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:15.822274 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.822302 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:44:15.822376 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:44:15.822510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":11,"thread_start_us":495,"threads_started":1}
I20260928 16:44:15.824352 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4201229ff82f4e499ee903969e29efaa" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" is_pre_election: true
I20260928 16:44:15.824352 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4201229ff82f4e499ee903969e29efaa" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" is_pre_election: true
I20260928 16:44:15.824952 2420 raft_consensus.cc:2454] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:15.824947 2706 raft_consensus.cc:2454] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:15.825744 2492 leader_election.cc:302] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:15.826253 2783 raft_consensus.cc:2781] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:15.826396 2783 raft_consensus.cc:482] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:15.826501 2783 raft_consensus.cc:3037] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:15.828881 2783 raft_consensus.cc:504] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.829557 2783 leader_election.cc:288] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:15.830240 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4201229ff82f4e499ee903969e29efaa" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
I20260928 16:44:15.830245 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4201229ff82f4e499ee903969e29efaa" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
I20260928 16:44:15.830457 2706 raft_consensus.cc:3037] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:15.830446 2420 raft_consensus.cc:3037] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:15.832791 2706 raft_consensus.cc:2454] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:15.832813 2420 raft_consensus.cc:2454] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:15.833302 2492 leader_election.cc:302] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, 6bd2a5839d7c45da96cce2861b05a5e0, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:15.833678 2783 raft_consensus.cc:2781] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:15.834097 2783 raft_consensus.cc:686] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Becoming Leader. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Running, Role: LEADER
I20260928 16:44:15.834666 2783 consensus_queue.cc:237] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.839581 2260 catalog_manager.cc:5495] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:15.863916 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:15.858280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.26.101.2:43737)
0928 16:44:15.858478 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:15.858483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:15.858503 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:15.858752 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:15.858757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:15.858853 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:15.858994 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:15.859395 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.860129 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.861530 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:15.861954 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:15.862048 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:15.862470 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:15.863051 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:15.863234 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:15.863489 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 16:44:15.863600 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:44:15.872501 2706 raft_consensus.cc:1261] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:15.872501 2420 raft_consensus.cc:1261] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:15.873647 2788 consensus_queue.cc:1035] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:15.874053 2783 consensus_queue.cc:1035] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:15.887642 2792 mvcc.cc:205] Tried to move back new op lower bound from 7334354353642889216 to 7334354353502642176. Current Snapshot: MvccSnapshot[applied={T|T < 7334354353642889216}]
I20260928 16:44:15.901935 2260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:15.902266 2260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:15.906535 2260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Sending DeleteTablet for 3 replicas of tablet 4201229ff82f4e499ee903969e29efaa
I20260928 16:44:15.907931 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4201229ff82f4e499ee903969e29efaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:15 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:58338
I20260928 16:44:15.907972 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 4201229ff82f4e499ee903969e29efaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:15 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260928 16:44:15.908245 2682 tablet_service.cc:1514] Processing DeleteTablet for tablet 4201229ff82f4e499ee903969e29efaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:15 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50668
I20260928 16:44:15.911888 2259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:15.912845 2259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:15.923455 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 (DEFAULT_TABLE table=test-table [id=5c0fef222b84452cb031067ba19f26a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.923765 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 (DEFAULT_TABLE table=test-table [id=5c0fef222b84452cb031067ba19f26a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.924141 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a7b79a63714aeaa44dd1f8fecab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.924173 2680 tablet_service.cc:1467] Processing CreateTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 (DEFAULT_TABLE table=test-table [id=5c0fef222b84452cb031067ba19f26a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:15.924429 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a7b79a63714aeaa44dd1f8fecab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.924696 2680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a7b79a63714aeaa44dd1f8fecab893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:15.925668 2803 tablet_replica.cc:330] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: stopping tablet replica
I20260928 16:44:15.926159 2803 raft_consensus.cc:2229] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:15.929340 2803 raft_consensus.cc:2258] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:15.931000 2778 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Bootstrap starting.
I20260928 16:44:15.932824 2778 tablet_bootstrap.cc:654] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.933511 2779 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Bootstrap starting.
I20260928 16:44:15.935055 2779 tablet_bootstrap.cc:654] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.939970 2777 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Bootstrap starting.
I20260928 16:44:15.941797 2777 tablet_bootstrap.cc:654] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:15.944296 2779 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: No bootstrap required, opened a new log
I20260928 16:44:15.944521 2779 ts_tablet_manager.cc:1397] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260928 16:44:15.945461 2779 raft_consensus.cc:348] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.945782 2779 raft_consensus.cc:374] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.945938 2779 raft_consensus.cc:729] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26649f9e6a894eaeba2bd1dcc7bc2da7, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.946380 2779 consensus_queue.cc:260] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.949733 2779 ts_tablet_manager.cc:1428] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 16:44:15.952883 2802 tablet_replica.cc:330] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: stopping tablet replica
I20260928 16:44:15.952958 2803 ts_tablet_manager.cc:1905] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:15.953446 2802 raft_consensus.cc:2229] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:15.954097 2801 tablet_replica.cc:330] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: stopping tablet replica
I20260928 16:44:15.954380 2802 raft_consensus.cc:2258] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:15.954622 2801 raft_consensus.cc:2229] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:15.955667 2778 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: No bootstrap required, opened a new log
I20260928 16:44:15.955902 2778 ts_tablet_manager.cc:1397] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.007s
I20260928 16:44:15.956846 2802 ts_tablet_manager.cc:1905] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:15.957706 2801 raft_consensus.cc:2258] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:15.958492 2778 raft_consensus.cc:348] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.958875 2778 raft_consensus.cc:374] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.959023 2778 raft_consensus.cc:729] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.959468 2778 consensus_queue.cc:260] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.963061 2801 ts_tablet_manager.cc:1905] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:15.963738 2778 ts_tablet_manager.cc:1428] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 16:44:15.964624 2777 tablet_bootstrap.cc:492] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: No bootstrap required, opened a new log
I20260928 16:44:15.964787 2777 ts_tablet_manager.cc:1397] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.016s
I20260928 16:44:15.965662 2777 raft_consensus.cc:348] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.966419 2802 ts_tablet_manager.cc:1918] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:15.966642 2802 log.cc:1196] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/4201229ff82f4e499ee903969e29efaa
I20260928 16:44:15.967252 2802 ts_tablet_manager.cc:1939] T 4201229ff82f4e499ee903969e29efaa P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting consensus metadata
I20260928 16:44:15.969225 2248 catalog_manager.cc:4841] TS 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611): tablet 4201229ff82f4e499ee903969e29efaa (table test-table [id=c672c8668ef1433b8f60a98887ee9c1a]) successfully deleted
I20260928 16:44:15.969362 2803 ts_tablet_manager.cc:1918] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:15.969528 2803 log.cc:1196] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/4201229ff82f4e499ee903969e29efaa
I20260928 16:44:15.969955 2803 ts_tablet_manager.cc:1939] T 4201229ff82f4e499ee903969e29efaa P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting consensus metadata
I20260928 16:44:15.971244 2777 raft_consensus.cc:374] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:15.971393 2777 raft_consensus.cc:729] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Initialized, Role: FOLLOWER
I20260928 16:44:15.971714 2777 consensus_queue.cc:260] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:15.972378 2777 ts_tablet_manager.cc:1428] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260928 16:44:15.973451 2247 catalog_manager.cc:4841] TS 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881): tablet 4201229ff82f4e499ee903969e29efaa (table test-table [id=c672c8668ef1433b8f60a98887ee9c1a]) successfully deleted
I20260928 16:44:15.976862 2801 ts_tablet_manager.cc:1918] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:15.977108 2801 log.cc:1196] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/4201229ff82f4e499ee903969e29efaa
I20260928 16:44:15.977583 2801 ts_tablet_manager.cc:1939] T 4201229ff82f4e499ee903969e29efaa P a39f6824d86e4ad98c89bae9f7a4d766: Deleting consensus metadata
I20260928 16:44:15.979063 2245 catalog_manager.cc:4841] TS a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737): tablet 4201229ff82f4e499ee903969e29efaa (table test-table [id=c672c8668ef1433b8f60a98887ee9c1a]) successfully deleted
I20260928 16:44:16.010633 2784 raft_consensus.cc:482] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:16.010867 2784 raft_consensus.cc:504] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.012638 2784 leader_election.cc:288] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.020416 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.013244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55809 (local address 127.26.101.2:43737)
0928 16:44:16.013413 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.013419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.013501 (+ 82us) server_negotiation.cc:407] Connection header received
0928 16:44:16.013810 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.013816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.013903 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.013970 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:16.014764 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.015983 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.018376 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.019311 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.019479 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.019559 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.019902 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.020057 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:44:16.020648 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.013345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53243 (local address 127.26.101.3:44881)
0928 16:44:16.013536 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.013542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.014061 (+ 519us) server_negotiation.cc:407] Connection header received
0928 16:44:16.014230 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.014236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.014327 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.014430 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:16.014998 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.015752 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.018321 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.019639 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.019809 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.019882 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.020221 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.020384 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:44:16.020679 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.012951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:44881 (local address 127.26.101.1:53243)
0928 16:44:16.013962 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.013979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.014087 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.014476 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.014481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.014508 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:16.014856 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.014865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.015880 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.015885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.018141 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.018159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.019847 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.019853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.020059 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.020089 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.020179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.020325 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":1,"thread_start_us":750,"threads_started":1}
I20260928 16:44:16.021509 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.012858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43737 (local address 127.26.101.1:55809)
0928 16:44:16.013378 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.013398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.013618 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.014186 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.014191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.014215 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:16.014556 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.014572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.016151 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.016157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.018182 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.018192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.019525 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.019531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.019688 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.019723 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.021043 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.021175 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":21,"thread_start_us":562,"threads_started":1}
I20260928 16:44:16.021574 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a7b79a63714aeaa44dd1f8fecab893" candidate_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" is_pre_election: true
I20260928 16:44:16.021860 2706 raft_consensus.cc:2454] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd2a5839d7c45da96cce2861b05a5e0 in term 0.
I20260928 16:44:16.022421 2351 leader_election.cc:302] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, 6bd2a5839d7c45da96cce2861b05a5e0; no voters:
I20260928 16:44:16.022902 2784 raft_consensus.cc:2781] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:16.023020 2784 raft_consensus.cc:482] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:16.023131 2784 raft_consensus.cc:3037] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.022894 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a7b79a63714aeaa44dd1f8fecab893" candidate_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" is_pre_election: true
I20260928 16:44:16.023450 2560 raft_consensus.cc:2454] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd2a5839d7c45da96cce2861b05a5e0 in term 0.
I20260928 16:44:16.025704 2784 raft_consensus.cc:504] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.026337 2784 leader_election.cc:288] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [CANDIDATE]: Term 1 election: Requested vote from peers a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.026819 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a7b79a63714aeaa44dd1f8fecab893" candidate_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f6824d86e4ad98c89bae9f7a4d766"
I20260928 16:44:16.027034 2560 raft_consensus.cc:3037] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.026952 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a7b79a63714aeaa44dd1f8fecab893" candidate_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
I20260928 16:44:16.027206 2706 raft_consensus.cc:3037] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.029582 2560 raft_consensus.cc:2454] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd2a5839d7c45da96cce2861b05a5e0 in term 1.
I20260928 16:44:16.030040 2349 leader_election.cc:302] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd2a5839d7c45da96cce2861b05a5e0, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:16.030227 2706 raft_consensus.cc:2454] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd2a5839d7c45da96cce2861b05a5e0 in term 1.
I20260928 16:44:16.030339 2784 raft_consensus.cc:2781] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:16.039188 2784 raft_consensus.cc:686] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Running, Role: LEADER
I20260928 16:44:16.039736 2784 consensus_queue.cc:237] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.043692 2259 catalog_manager.cc:5495] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:44:16 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613852, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.62@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:16.098349 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.081841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40427 (local address 127.0.0.1:46918)
0928 16:44:16.082540 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.082560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.082725 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.083069 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.083104 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.084051 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:16.084413 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.084427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.085435 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.085441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.087029 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.087045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.087825 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.087833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.088009 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.088762 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:16.088803 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:16.091576 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:16.095481 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:16.095497 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:16.095503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:16.095930 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:16.096446 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:16.096450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:16.096453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:16.096614 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:16.097272 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:16.097281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:16.097464 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.097602 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.097892 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":363,"threads_started":1}
I20260928 16:44:16.098387 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.081810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.26.101.62:40427)
0928 16:44:16.082052 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.082058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.082655 (+ 597us) server_negotiation.cc:407] Connection header received
0928 16:44:16.082846 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.082850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.082917 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.083006 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:16.084564 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.085302 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.087196 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.087630 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.087723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.091794 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:16.091801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:16.091806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:16.091847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:16.095317 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:16.096080 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:16.096086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:16.096090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:16.096247 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:16.096777 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:16.096782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:16.096785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:16.097134 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:16.097282 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.097724 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.098051 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":109}
I20260928 16:44:16.104339 2259 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:46918:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:44:16.105465 2259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:16.112512 2400 tablet_service.cc:1467] Processing CreateTablet for tablet b28178facdb044e6b966f2c6f078cd62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f726856902cc4613901a2133f4ae85a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:16.112649 2540 tablet_service.cc:1467] Processing CreateTablet for tablet b28178facdb044e6b966f2c6f078cd62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f726856902cc4613901a2133f4ae85a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:16.113148 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28178facdb044e6b966f2c6f078cd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.113108 2680 tablet_service.cc:1467] Processing CreateTablet for tablet b28178facdb044e6b966f2c6f078cd62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f726856902cc4613901a2133f4ae85a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:44:16.113296 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28178facdb044e6b966f2c6f078cd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.113591 2680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28178facdb044e6b966f2c6f078cd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.120833 2779 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Bootstrap starting.
I20260928 16:44:16.120836 2777 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: Bootstrap starting.
I20260928 16:44:16.120975 2778 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: Bootstrap starting.
I20260928 16:44:16.122705 2777 tablet_bootstrap.cc:654] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.122705 2779 tablet_bootstrap.cc:654] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.122763 2778 tablet_bootstrap.cc:654] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.132573 2778 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: No bootstrap required, opened a new log
I20260928 16:44:16.132874 2778 ts_tablet_manager.cc:1397] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260928 16:44:16.133680 2778 raft_consensus.cc:348] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.134032 2778 raft_consensus.cc:374] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.134152 2778 raft_consensus.cc:729] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.134375 2779 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: No bootstrap required, opened a new log
I20260928 16:44:16.134473 2778 consensus_queue.cc:260] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.134634 2779 ts_tablet_manager.cc:1397] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260928 16:44:16.134984 2784 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.135288 2784 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.135425 2779 raft_consensus.cc:348] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.135668 2779 raft_consensus.cc:374] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.135740 2779 raft_consensus.cc:729] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26649f9e6a894eaeba2bd1dcc7bc2da7, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.135987 2779 consensus_queue.cc:260] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.136335 2777 tablet_bootstrap.cc:492] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: No bootstrap required, opened a new log
I20260928 16:44:16.136548 2777 ts_tablet_manager.cc:1397] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260928 16:44:16.136427 2785 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.136811 2785 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.137332 2777 raft_consensus.cc:348] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.137625 2777 raft_consensus.cc:374] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.137715 2777 raft_consensus.cc:729] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.137777 2779 ts_tablet_manager.cc:1428] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:16.137972 2777 consensus_queue.cc:260] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.138614 2777 ts_tablet_manager.cc:1428] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 16:44:16.138427 2796 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.138849 2796 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 16:44:16.139724 2471 tablet.cc:2369] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:16.141625 2778 ts_tablet_manager.cc:1428] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20260928 16:44:16.182830 2612 tablet.cc:2369] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:16.377120 2796 raft_consensus.cc:482] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:16.377334 2796 raft_consensus.cc:504] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.378043 2796 leader_election.cc:288] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.378604 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28178facdb044e6b966f2c6f078cd62" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" is_pre_election: true
I20260928 16:44:16.378607 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28178facdb044e6b966f2c6f078cd62" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" is_pre_election: true
I20260928 16:44:16.378837 2420 raft_consensus.cc:2454] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:16.378844 2706 raft_consensus.cc:2454] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:16.379292 2492 leader_election.cc:302] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, 6bd2a5839d7c45da96cce2861b05a5e0, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:16.379556 2796 raft_consensus.cc:2781] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:16.379670 2796 raft_consensus.cc:482] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:16.379758 2796 raft_consensus.cc:3037] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.381776 2796 raft_consensus.cc:504] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.382318 2796 leader_election.cc:288] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.382804 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28178facdb044e6b966f2c6f078cd62" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
I20260928 16:44:16.382835 2706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28178facdb044e6b966f2c6f078cd62" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
I20260928 16:44:16.383010 2420 raft_consensus.cc:3037] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.383033 2706 raft_consensus.cc:3037] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.385128 2706 raft_consensus.cc:2454] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:16.385522 2492 leader_election.cc:302] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:16.385792 2796 raft_consensus.cc:2781] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:16.385955 2796 raft_consensus.cc:686] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Becoming Leader. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Running, Role: LEADER
I20260928 16:44:16.386016 2420 raft_consensus.cc:2454] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:16.386281 2796 consensus_queue.cc:237] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.387873 2788 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: TxnStatusTablet state changed. Reason: New leader a39f6824d86e4ad98c89bae9f7a4d766. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.388128 2788 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:16.388614 2821 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:44:16.388832 2821 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:44:16.392926 2260 catalog_manager.cc:5495] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:16.440201 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.435032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43737 (local address 127.0.0.1:43940)
0928 16:44:16.435294 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.435315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.435485 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.435902 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.435907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.435919 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:16.436090 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.436099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.437195 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.437201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.438226 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.438239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.438980 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.438986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.439174 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.439237 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.439651 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.439674 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.439744 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.439897 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 16:44:16.440201 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.435141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.26.101.2:43737)
0928 16:44:16.435325 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.435334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.435371 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:44:16.435710 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.435714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.435784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.435859 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:16.436231 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.437067 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.438374 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.438829 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.438936 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.439420 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.439532 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.439609 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.439799 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.439884 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:44:16.444401 2706 raft_consensus.cc:1261] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:16.444515 2420 raft_consensus.cc:1261] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:16.445112 2788 consensus_queue.cc:1035] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.445601 2796 consensus_queue.cc:1035] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.448724 2809 consensus_queue.cc:1035] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.448940 2784 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: TxnStatusTablet state changed. Reason: New leader a39f6824d86e4ad98c89bae9f7a4d766. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.449169 2784 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.450610 2823 mvcc.cc:205] Tried to move back new op lower bound from 7334354355989979136 to 7334354355761041408. Current Snapshot: MvccSnapshot[applied={T|T < 7334354355989979136}]
I20260928 16:44:16.453415 2809 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.453751 2809 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 6bd2a5839d7c45da96cce2861b05a5e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.456669 2824 mvcc.cc:205] Tried to move back new op lower bound from 7334354355989979136 to 7334354355761041408. Current Snapshot: MvccSnapshot[applied={T|T < 7334354355989979136 or (T in {7334354355989979136})}]
I20260928 16:44:16.462579 2809 consensus_queue.cc:1035] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.472216 2785 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: TxnStatusTablet state changed. Reason: New leader a39f6824d86e4ad98c89bae9f7a4d766. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.472613 2785 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.494297 2788 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.494603 2788 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:16.496917 2788 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.497165 2788 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P a39f6824d86e4ad98c89bae9f7a4d766: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:44:16.499342 2785 tablet_replica.cc:439] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } } }
I20260928 16:44:16.499596 2785 tablet_replica.cc:442] T b28178facdb044e6b966f2c6f078cd62 P 26649f9e6a894eaeba2bd1dcc7bc2da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:44:16.524312 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.503483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.26.101.1:41611)
0928 16:44:16.518371 (+ 14888us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.518376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.518398 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:16.518494 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.518498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.518576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.518744 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:16.519318 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.520074 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.521816 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.522390 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.522504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.523018 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.523678 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.523823 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.523925 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.524009 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14783,"thread_start_us":14609,"threads_started":1}
I20260928 16:44:16.539923 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.534639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45097 (local address 127.26.101.2:43737)
0928 16:44:16.534841 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.534847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.535169 (+ 322us) server_negotiation.cc:407] Connection header received
0928 16:44:16.535384 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.535390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.535463 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.535548 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:16.535961 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.536733 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.538080 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.538525 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.538636 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.539129 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.539263 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.539338 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.539479 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.539559 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:44:16.539958 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.534504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43737 (local address 127.26.101.1:45097)
0928 16:44:16.535038 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.535053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.535241 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.535574 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.535578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.535590 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:16.535814 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.535823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.536859 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.536865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.537926 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.537938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.538687 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.538694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.538870 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.538927 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.539362 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.539375 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.539443 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.539605 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260928 16:44:16.578022 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.571198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36669 (local address 127.26.101.1:41611)
0928 16:44:16.571372 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.571378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.572050 (+ 672us) server_negotiation.cc:407] Connection header received
0928 16:44:16.572233 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.572237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.572309 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.572379 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:16.573713 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.574540 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.576133 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.576646 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.576757 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.577245 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.577385 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.577489 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.577644 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.577731 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:44:16.578071 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.571246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41611 (local address 127.26.101.2:36669)
0928 16:44:16.571958 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:16.571978 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:16.572128 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:16.572431 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:16.572436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:16.572450 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:16.573540 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.573557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.574667 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.574673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.575988 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:16.576006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.576802 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.576808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:16.576967 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.577046 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.577508 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.577525 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:44:16.577621 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:44:16.577761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":364,"threads_started":1}
I20260928 16:44:16.612744 2260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:16.613030 2260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:16.616539 2260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Sending DeleteTablet for 3 replicas of tablet 94a7b79a63714aeaa44dd1f8fecab893
I20260928 16:44:16.617336 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:58338
I20260928 16:44:16.617897 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260928 16:44:16.619508 2680 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a7b79a63714aeaa44dd1f8fecab893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50668
I20260928 16:44:16.629882 2847 tablet_replica.cc:330] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: stopping tablet replica
I20260928 16:44:16.630270 2847 raft_consensus.cc:2229] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:16.630985 2847 raft_consensus.cc:2258] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:16.632256 2847 ts_tablet_manager.cc:1905] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.633162 2849 tablet_replica.cc:330] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: stopping tablet replica
I20260928 16:44:16.638599 2849 raft_consensus.cc:2229] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:16.638990 2849 raft_consensus.cc:2258] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:16.640247 2849 ts_tablet_manager.cc:1905] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.640544 2847 ts_tablet_manager.cc:1918] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:16.640741 2847 log.cc:1196] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/94a7b79a63714aeaa44dd1f8fecab893
I20260928 16:44:16.641289 2847 ts_tablet_manager.cc:1939] T 94a7b79a63714aeaa44dd1f8fecab893 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting consensus metadata
I20260928 16:44:16.642572 2248 catalog_manager.cc:4841] TS 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611): tablet 94a7b79a63714aeaa44dd1f8fecab893 (table test-table [id=5c0fef222b84452cb031067ba19f26a6]) successfully deleted
I20260928 16:44:16.644200 2848 tablet_replica.cc:330] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: stopping tablet replica
I20260928 16:44:16.644642 2848 raft_consensus.cc:2229] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:16.644984 2848 raft_consensus.cc:2258] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:16.646071 2848 ts_tablet_manager.cc:1905] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.654955 2848 ts_tablet_manager.cc:1918] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:16.655241 2848 log.cc:1196] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/94a7b79a63714aeaa44dd1f8fecab893
I20260928 16:44:16.655766 2848 ts_tablet_manager.cc:1939] T 94a7b79a63714aeaa44dd1f8fecab893 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting consensus metadata
I20260928 16:44:16.657181 2849 ts_tablet_manager.cc:1918] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:16.657130 2245 catalog_manager.cc:4841] TS a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737): tablet 94a7b79a63714aeaa44dd1f8fecab893 (table test-table [id=5c0fef222b84452cb031067ba19f26a6]) successfully deleted
I20260928 16:44:16.657392 2849 log.cc:1196] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/94a7b79a63714aeaa44dd1f8fecab893
I20260928 16:44:16.657845 2849 ts_tablet_manager.cc:1939] T 94a7b79a63714aeaa44dd1f8fecab893 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting consensus metadata
I20260928 16:44:16.658913 2247 catalog_manager.cc:4841] TS 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881): tablet 94a7b79a63714aeaa44dd1f8fecab893 (table test-table [id=5c0fef222b84452cb031067ba19f26a6]) successfully deleted
Sep 28 16:44:16 dist-test-slave-t0tw krb5kdc[2215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:44:16 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:16 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:16.723673 2858 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:16.689815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.26.101.62:40427)
0928 16:44:16.705750 (+ 15935us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.705758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.705782 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:44:16.705852 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.705858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.705949 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.706067 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:16.708115 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.709286 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.711255 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.711971 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.712105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.723333 (+ 11228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46930: BlockingRecv error: recv got EOF from 127.0.0.1:46930 (error 108)
Metrics: {"server-negotiator.queue_time_us":15761,"thread_start_us":16140,"threads_started":1}
Sep 28 16:44:16 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM
I20260928 16:44:16.746585 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.725560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.26.101.62:40427)
0928 16:44:16.725741 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.725748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.727638 (+ 1890us) server_negotiation.cc:407] Connection header received
0928 16:44:16.727825 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.727831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.727933 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.728054 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:16.729705 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.730809 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.732716 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.733381 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.733542 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.738942 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:16.738950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:16.738955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:16.739008 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:16.742614 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:16.743591 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:16.743598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:16.743602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:16.743787 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:16.744256 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:16.744261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:16.744265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:16.744623 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:16.744974 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.745971 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.746212 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:44:16.750946 2260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:16.751742 2260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:16.759550 2400 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c (DEFAULT_TABLE table=test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:16.759773 2680 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c (DEFAULT_TABLE table=test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:16.760061 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6dbfd8d3541dabac9d452f2d69c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.760299 2680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6dbfd8d3541dabac9d452f2d69c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.764060 2540 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c (DEFAULT_TABLE table=test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:16.764511 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6dbfd8d3541dabac9d452f2d69c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:16.783658 2865 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Bootstrap starting.
I20260928 16:44:16.785820 2865 tablet_bootstrap.cc:654] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.796566 2865 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: No bootstrap required, opened a new log
I20260928 16:44:16.796764 2865 ts_tablet_manager.cc:1397] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260928 16:44:16.797706 2865 raft_consensus.cc:348] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.798002 2865 raft_consensus.cc:374] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.798099 2865 raft_consensus.cc:729] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.798349 2865 consensus_queue.cc:260] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.800015 2865 ts_tablet_manager.cc:1428] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:16.806998 2867 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Bootstrap starting.
I20260928 16:44:16.807361 2864 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Bootstrap starting.
I20260928 16:44:16.809422 2864 tablet_bootstrap.cc:654] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.812093 2867 tablet_bootstrap.cc:654] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:16.824072 2864 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: No bootstrap required, opened a new log
I20260928 16:44:16.825014 2867 tablet_bootstrap.cc:492] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: No bootstrap required, opened a new log
I20260928 16:44:16.825244 2867 ts_tablet_manager.cc:1397] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20260928 16:44:16.826107 2867 raft_consensus.cc:348] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.826550 2864 ts_tablet_manager.cc:1397] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260928 16:44:16.827732 2867 raft_consensus.cc:374] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.827903 2867 raft_consensus.cc:729] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26649f9e6a894eaeba2bd1dcc7bc2da7, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.827456 2864 raft_consensus.cc:348] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.828228 2867 consensus_queue.cc:260] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.828995 2867 ts_tablet_manager.cc:1428] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 16:44:16.829239 2864 raft_consensus.cc:374] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:16.829411 2864 raft_consensus.cc:729] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Initialized, Role: FOLLOWER
I20260928 16:44:16.829737 2864 consensus_queue.cc:260] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.830513 2864 ts_tablet_manager.cc:1428] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 16:44:16.914747 2842 raft_consensus.cc:482] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:16.914978 2842 raft_consensus.cc:504] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.915782 2842 leader_election.cc:288] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.916450 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6dbfd8d3541dabac9d452f2d69c6c" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" is_pre_election: true
I20260928 16:44:16.916450 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6dbfd8d3541dabac9d452f2d69c6c" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" is_pre_election: true
I20260928 16:44:16.916745 2420 raft_consensus.cc:2454] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:16.916734 2705 raft_consensus.cc:2454] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:16.917301 2492 leader_election.cc:302] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, 6bd2a5839d7c45da96cce2861b05a5e0, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:16.917585 2842 raft_consensus.cc:2781] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:16.917704 2842 raft_consensus.cc:482] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:16.917809 2842 raft_consensus.cc:3037] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.919989 2842 raft_consensus.cc:504] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.921180 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6dbfd8d3541dabac9d452f2d69c6c" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
I20260928 16:44:16.921213 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6dbfd8d3541dabac9d452f2d69c6c" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
I20260928 16:44:16.921424 2705 raft_consensus.cc:3037] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.921439 2420 raft_consensus.cc:3037] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:16.924494 2705 raft_consensus.cc:2454] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:16.924621 2420 raft_consensus.cc:2454] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:16.925012 2842 leader_election.cc:288] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:16.925055 2492 leader_election.cc:302] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:16.925371 2842 raft_consensus.cc:2781] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:16.925546 2842 raft_consensus.cc:686] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Becoming Leader. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Running, Role: LEADER
I20260928 16:44:16.925848 2842 consensus_queue.cc:237] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:16.931599 2260 catalog_manager.cc:5495] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:16.948978 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:16.943891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.26.101.2:43737)
0928 16:44:16.944090 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:16.944096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:16.944119 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:16.944189 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:16.944195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:16.944280 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:16.944379 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:16.944852 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.945742 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.947248 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:16.947708 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:16.947837 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:16.948230 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:16.948370 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:16.948445 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:16.948592 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 16:44:16.948675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:44:16.951896 2420 raft_consensus.cc:1261] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:16.952633 2788 consensus_queue.cc:1035] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.953370 2705 raft_consensus.cc:1261] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:16.955722 2788 consensus_queue.cc:1035] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:16.976864 2260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:16.977084 2260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:16.980368 2260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Sending DeleteTablet for 3 replicas of tablet b3e6dbfd8d3541dabac9d452f2d69c6c
I20260928 16:44:16.981340 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260928 16:44:16.981345 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:58338
I20260928 16:44:16.981667 2848 tablet_replica.cc:330] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: stopping tablet replica
I20260928 16:44:16.981760 2847 tablet_replica.cc:330] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: stopping tablet replica
I20260928 16:44:16.981873 2848 raft_consensus.cc:2229] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:16.981966 2847 raft_consensus.cc:2229] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:16.982192 2847 raft_consensus.cc:2258] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:16.982215 2848 raft_consensus.cc:2258] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:16.983453 2847 ts_tablet_manager.cc:1905] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.983616 2680 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6dbfd8d3541dabac9d452f2d69c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:16 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50668
I20260928 16:44:16.983997 2849 tablet_replica.cc:330] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: stopping tablet replica
I20260928 16:44:16.983850 2260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:44:16.984208 2849 raft_consensus.cc:2229] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:16.984453 2849 raft_consensus.cc:2258] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 16:44:16.984798 2260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:16.985546 2848 ts_tablet_manager.cc:1905] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.985566 2849 ts_tablet_manager.cc:1905] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:16.991966 2848 ts_tablet_manager.cc:1918] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:16.991966 2847 ts_tablet_manager.cc:1918] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:16.991988 2849 ts_tablet_manager.cc:1918] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:16.992142 2849 log.cc:1196] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/b3e6dbfd8d3541dabac9d452f2d69c6c
I20260928 16:44:16.992152 2848 log.cc:1196] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/b3e6dbfd8d3541dabac9d452f2d69c6c
I20260928 16:44:16.992244 2847 log.cc:1196] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/b3e6dbfd8d3541dabac9d452f2d69c6c
I20260928 16:44:16.992571 2848 ts_tablet_manager.cc:1939] T b3e6dbfd8d3541dabac9d452f2d69c6c P a39f6824d86e4ad98c89bae9f7a4d766: Deleting consensus metadata
I20260928 16:44:16.992576 2849 ts_tablet_manager.cc:1939] T b3e6dbfd8d3541dabac9d452f2d69c6c P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting consensus metadata
I20260928 16:44:16.992753 2847 ts_tablet_manager.cc:1939] T b3e6dbfd8d3541dabac9d452f2d69c6c P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting consensus metadata
I20260928 16:44:16.993887 2248 catalog_manager.cc:4841] TS 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611): tablet b3e6dbfd8d3541dabac9d452f2d69c6c (table test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]) successfully deleted
I20260928 16:44:16.993966 2247 catalog_manager.cc:4841] TS 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881): tablet b3e6dbfd8d3541dabac9d452f2d69c6c (table test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]) successfully deleted
I20260928 16:44:16.993865 2245 catalog_manager.cc:4841] TS a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737): tablet b3e6dbfd8d3541dabac9d452f2d69c6c (table test-table [id=a3505bbbdacf4a77bb0ecdbfb411f7a8]) successfully deleted
I20260928 16:44:17.000281 2680 tablet_service.cc:1467] Processing CreateTablet for tablet d97806a5c994441e961427b48a9c1c00 (DEFAULT_TABLE table=test-table [id=0b9a8b20144b437dabcdb85fd02195ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:17.000250 2400 tablet_service.cc:1467] Processing CreateTablet for tablet d97806a5c994441e961427b48a9c1c00 (DEFAULT_TABLE table=test-table [id=0b9a8b20144b437dabcdb85fd02195ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:17.000759 2680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97806a5c994441e961427b48a9c1c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:17.000814 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97806a5c994441e961427b48a9c1c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:17.005038 2540 tablet_service.cc:1467] Processing CreateTablet for tablet d97806a5c994441e961427b48a9c1c00 (DEFAULT_TABLE table=test-table [id=0b9a8b20144b437dabcdb85fd02195ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:17.005517 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97806a5c994441e961427b48a9c1c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:17.006455 2867 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Bootstrap starting.
I20260928 16:44:17.008128 2864 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Bootstrap starting.
I20260928 16:44:17.008227 2867 tablet_bootstrap.cc:654] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:17.009922 2864 tablet_bootstrap.cc:654] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:17.012357 2865 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Bootstrap starting.
I20260928 16:44:17.014168 2865 tablet_bootstrap.cc:654] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:17.018802 2867 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: No bootstrap required, opened a new log
I20260928 16:44:17.019102 2867 ts_tablet_manager.cc:1397] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260928 16:44:17.020041 2867 raft_consensus.cc:348] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.020470 2867 raft_consensus.cc:374] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:17.020619 2867 raft_consensus.cc:729] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26649f9e6a894eaeba2bd1dcc7bc2da7, State: Initialized, Role: FOLLOWER
I20260928 16:44:17.020697 2864 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: No bootstrap required, opened a new log
I20260928 16:44:17.020905 2864 ts_tablet_manager.cc:1397] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260928 16:44:17.021073 2867 consensus_queue.cc:260] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.021879 2864 raft_consensus.cc:348] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.022172 2864 raft_consensus.cc:374] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:17.022279 2864 raft_consensus.cc:729] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2a5839d7c45da96cce2861b05a5e0, State: Initialized, Role: FOLLOWER
I20260928 16:44:17.022665 2867 ts_tablet_manager.cc:1428] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:44:17.022642 2864 consensus_queue.cc:260] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.023223 2864 ts_tablet_manager.cc:1428] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:44:17.028131 2865 tablet_bootstrap.cc:492] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: No bootstrap required, opened a new log
I20260928 16:44:17.028400 2865 ts_tablet_manager.cc:1397] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260928 16:44:17.029227 2865 raft_consensus.cc:348] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.029465 2865 raft_consensus.cc:374] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:17.029542 2865 raft_consensus.cc:729] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Initialized, Role: FOLLOWER
I20260928 16:44:17.029779 2865 consensus_queue.cc:260] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.030260 2865 ts_tablet_manager.cc:1428] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 16:44:17.122999 2788 raft_consensus.cc:482] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:17.123258 2788 raft_consensus.cc:504] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.123984 2788 leader_election.cc:288] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:17.124670 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97806a5c994441e961427b48a9c1c00" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" is_pre_election: true
I20260928 16:44:17.124730 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97806a5c994441e961427b48a9c1c00" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" is_pre_election: true
I20260928 16:44:17.124975 2705 raft_consensus.cc:2454] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:17.124979 2420 raft_consensus.cc:2454] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 0.
I20260928 16:44:17.125454 2493 leader_election.cc:302] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd2a5839d7c45da96cce2861b05a5e0, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:17.125756 2788 raft_consensus.cc:2781] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:17.125875 2788 raft_consensus.cc:482] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:17.125968 2788 raft_consensus.cc:3037] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:17.128041 2788 raft_consensus.cc:504] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.128623 2788 leader_election.cc:288] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611), 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881)
I20260928 16:44:17.129144 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97806a5c994441e961427b48a9c1c00" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd2a5839d7c45da96cce2861b05a5e0"
I20260928 16:44:17.129227 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97806a5c994441e961427b48a9c1c00" candidate_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7"
I20260928 16:44:17.129376 2420 raft_consensus.cc:3037] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:17.129427 2705 raft_consensus.cc:3037] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:17.132326 2705 raft_consensus.cc:2454] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:17.132406 2420 raft_consensus.cc:2454] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39f6824d86e4ad98c89bae9f7a4d766 in term 1.
I20260928 16:44:17.132820 2492 leader_election.cc:302] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26649f9e6a894eaeba2bd1dcc7bc2da7, a39f6824d86e4ad98c89bae9f7a4d766; no voters:
I20260928 16:44:17.133179 2788 raft_consensus.cc:2781] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:17.133301 2788 raft_consensus.cc:686] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Becoming Leader. State: Replica: a39f6824d86e4ad98c89bae9f7a4d766, State: Running, Role: LEADER
I20260928 16:44:17.133559 2788 consensus_queue.cc:237] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } }
I20260928 16:44:17.136123 2260 catalog_manager.cc:5495] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39f6824d86e4ad98c89bae9f7a4d766" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
I20260928 16:44:17.175544 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:17.170490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:42827 (local address 127.26.101.2:43737)
0928 16:44:17.170648 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:17.170656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:17.170929 (+ 273us) server_negotiation.cc:407] Connection header received
0928 16:44:17.171151 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:17.171154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:17.171223 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:17.171363 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:17.171778 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.172606 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.173907 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.174373 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.174466 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:17.174835 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:17.174953 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:17.175027 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:17.175177 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:44:17.175244 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:44:17.175593 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:17.170307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43737 (local address 127.26.101.2:42827)
0928 16:44:17.170839 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:17.170856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:17.171014 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:17.171433 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:17.171436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:17.171447 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:44:17.171666 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:17.171674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.172749 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.172754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:17.173766 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:17.173778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.174505 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.174509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:17.174643 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:17.174678 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:44:17.175039 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:44:17.175047 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:44:17.175144 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 16:44:17.175258 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":3,"thread_start_us":283,"threads_started":1}
I20260928 16:44:17.183158 2420 raft_consensus.cc:1261] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:17.183221 2705 raft_consensus.cc:1261] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Refusing update from remote peer a39f6824d86e4ad98c89bae9f7a4d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:17.183672 2788 consensus_queue.cc:1035] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd2a5839d7c45da96cce2861b05a5e0" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:17.183950 2796 consensus_queue.cc:1035] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26649f9e6a894eaeba2bd1dcc7bc2da7" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:17.229825 2260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:17.230088 2260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:17.233201 2260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d319fe489ee407cafb5ea40e3a88fe2: Sending DeleteTablet for 3 replicas of tablet d97806a5c994441e961427b48a9c1c00
I20260928 16:44:17.234138 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet d97806a5c994441e961427b48a9c1c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:17 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:58338
I20260928 16:44:17.234246 2680 tablet_service.cc:1514] Processing DeleteTablet for tablet d97806a5c994441e961427b48a9c1c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:17 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:50668
I20260928 16:44:17.234277 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet d97806a5c994441e961427b48a9c1c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:17 UTC) from {username='oryx', principal='oryx/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260928 16:44:17.234531 2849 tablet_replica.cc:330] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: stopping tablet replica
I20260928 16:44:17.234645 2847 tablet_replica.cc:330] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: stopping tablet replica
I20260928 16:44:17.234714 2849 raft_consensus.cc:2229] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:17.234828 2847 raft_consensus.cc:2229] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:17.234928 2849 raft_consensus.cc:2258] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:17.235033 2847 raft_consensus.cc:2258] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:17.235368 2848 tablet_replica.cc:330] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: stopping tablet replica
I20260928 16:44:17.235574 2848 raft_consensus.cc:2229] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:17.235931 2848 raft_consensus.cc:2258] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:17.236204 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 2332
I20260928 16:44:17.236995 2849 ts_tablet_manager.cc:1905] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:17.237289 2847 ts_tablet_manager.cc:1905] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:17.243956 2848 ts_tablet_manager.cc:1905] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:17.244928 2847 ts_tablet_manager.cc:1918] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:17.244930 2849 ts_tablet_manager.cc:1918] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:17.245074 2847 log.cc:1196] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/d97806a5c994441e961427b48a9c1c00
I20260928 16:44:17.245213 2849 log.cc:1196] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/d97806a5c994441e961427b48a9c1c00
I20260928 16:44:17.245663 2849 ts_tablet_manager.cc:1939] T d97806a5c994441e961427b48a9c1c00 P 26649f9e6a894eaeba2bd1dcc7bc2da7: Deleting consensus metadata
I20260928 16:44:17.245653 2847 ts_tablet_manager.cc:1939] T d97806a5c994441e961427b48a9c1c00 P 6bd2a5839d7c45da96cce2861b05a5e0: Deleting consensus metadata
I20260928 16:44:17.246956 2248 catalog_manager.cc:4841] TS 6bd2a5839d7c45da96cce2861b05a5e0 (127.26.101.1:41611): tablet d97806a5c994441e961427b48a9c1c00 (table test-table [id=0b9a8b20144b437dabcdb85fd02195ad]) successfully deleted
I20260928 16:44:17.246956 2247 catalog_manager.cc:4841] TS 26649f9e6a894eaeba2bd1dcc7bc2da7 (127.26.101.3:44881): tablet d97806a5c994441e961427b48a9c1c00 (table test-table [id=0b9a8b20144b437dabcdb85fd02195ad]) successfully deleted
I20260928 16:44:17.250645 2848 ts_tablet_manager.cc:1918] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:44:17.250771 2848 log.cc:1196] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/d97806a5c994441e961427b48a9c1c00
I20260928 16:44:17.251230 2848 ts_tablet_manager.cc:1939] T d97806a5c994441e961427b48a9c1c00 P a39f6824d86e4ad98c89bae9f7a4d766: Deleting consensus metadata
I20260928 16:44:17.252388 2245 catalog_manager.cc:4841] TS a39f6824d86e4ad98c89bae9f7a4d766 (127.26.101.2:43737): tablet d97806a5c994441e961427b48a9c1c00 (table test-table [id=0b9a8b20144b437dabcdb85fd02195ad]) successfully deleted
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.1@KRBTEST.COM
I20260928 16:44:17.265841 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:17.237095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.26.101.1:41611)
0928 16:44:17.251556 (+ 14461us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:17.251562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:17.251585 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:17.251650 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:17.251654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:17.251744 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:17.251877 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:17.253180 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.253978 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.255272 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.255692 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.255792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:17.259779 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:17.259796 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:17.259813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:17.259849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:17.263169 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.263947 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.263955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.263958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.264114 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.264430 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.264433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.264435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.264778 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:17.265014 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:17.265274 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:44:17.265462 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14330,"thread_start_us":14141,"threads_started":1}
I20260928 16:44:17.266513 2465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:17.389715 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 2332
I20260928 16:44:17.409921 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 2474
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.2@KRBTEST.COM
I20260928 16:44:17.425365 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:17.410890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.26.101.2:43737)
0928 16:44:17.411050 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:17.411054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:17.411097 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:44:17.411199 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:17.411203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:17.411272 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:17.411435 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:17.412682 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.413449 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.414758 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.415341 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.415445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:17.419676 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:17.419698 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:17.419719 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:17.419772 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:17.422612 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.423565 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.423572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.423575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.423724 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.424050 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.424053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.424055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.424440 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:17.424662 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:17.424908 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 16:44:17.425118 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:44:17.426000 2606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:17.577508 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 2474
I20260928 16:44:17.601454 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 2615
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.26.101.3@KRBTEST.COM
I20260928 16:44:17.617539 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:17.602236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.26.101.3:44881)
0928 16:44:17.602717 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:17.602722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:17.602738 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:17.602792 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:17.602796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:17.602865 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:17.602963 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:17.604330 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.605104 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.606454 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:17.606910 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:17.607024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:17.611625 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:17.611642 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:17.611660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:17.611701 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:17.614718 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.615578 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.615586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.615588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.615754 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:17.616133 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:17.616137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:17.616140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:17.616546 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:17.616793 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:17.617020 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 16:44:17.617215 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260928 16:44:17.618216 2747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:17.716533 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 2615
I20260928 16:44:17.733212 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 2229
I20260928 16:44:17.734462 2301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:17.853304 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 2229
2026-09-28T16:44:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6769 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2893](info): set up 2 sockets
Sep 28 16:44:17 dist-test-slave-t0tw krb5kdc[2893](info): commencing operation
krb5kdc: starting...
W20260928 16:44:18.887530 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:19 dist-test-slave-t0tw krb5kdc[2893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:44:19.118433 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44201
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:19.448249 2906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:19.448614 2906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:19.448750 2906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:19.456636 2906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:19.456805 2906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:19.456879 2906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:19.456959 2906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:19.457028 2906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:19.467325 2906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:19.468721 2906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:19.470082 2906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:19.482424 2911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:19.482340 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:19.482458 2912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:19.483534 2906 server_base.cc:1034] running on GCE node
I20260928 16:44:19.484342 2906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:19.485867 2906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:19.487138 2906 hybrid_clock.cc:648] HybridClock initialized: now 1790613859487056 us; error 35 us; skew 500 ppm
I20260928 16:44:19.487561 2906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T16:44:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1702 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:19 dist-test-slave-t0tw krb5kdc[2920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:19 dist-test-slave-t0tw krb5kdc[2920](info): set up 2 sockets
Sep 28 16:44:19 dist-test-slave-t0tw krb5kdc[2920](info): commencing operation
krb5kdc: starting...
W20260928 16:44:20.603643 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:20 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:44:20.834669 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46567
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.166340 2933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.166661 2933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.166801 2933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.174228 2933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:21.174377 2933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.174451 2933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.174525 2933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:21.174587 2933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:21.185068 2933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:21.186478 2933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.187939 2933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.199965 2939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.200037 2938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.200003 2941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.200997 2933 server_base.cc:1034] running on GCE node
I20260928 16:44:21.202123 2933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.204015 2933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.205245 2933 hybrid_clock.cc:648] HybridClock initialized: now 1790613861205193 us; error 35 us; skew 500 ppm
I20260928 16:44:21.205647 2933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:21 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.214834 2933 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:44:21.216609 2933 webserver.cc:460] Webserver started at http://127.26.101.62:36815/ using document root <none> and password file <none>
I20260928 16:44:21.217361 2933 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.217489 2933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.217823 2933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.221016 2933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "25e108258c84499593b751de17f24653"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f6a37a885b756d0288bbb07f28ad2026"
server_key_iv: "40a7a5661257d5cd03173a87b5d11072"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.221809 2933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "25e108258c84499593b751de17f24653"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f6a37a885b756d0288bbb07f28ad2026"
server_key_iv: "40a7a5661257d5cd03173a87b5d11072"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.227440 2933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:44:21.231262 2948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.232744 2933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:21.232939 2933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "25e108258c84499593b751de17f24653"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f6a37a885b756d0288bbb07f28ad2026"
server_key_iv: "40a7a5661257d5cd03173a87b5d11072"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.233146 2933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.257009 2933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.261726 2933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.262032 2933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.281230 2933 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:46567
I20260928 16:44:21.281242 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:46567 every 8 connection(s)
I20260928 16:44:21.282666 2933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:21.286460 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:21.287300 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 2933
I20260928 16:44:21.287487 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:21.288127 27028 external_mini_cluster.cc:1428] Setting key dc8950a2715f4728a2919a5502870a0c
I20260928 16:44:21.300689 3002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Bootstrap starting.
Sep 28 16:44:21 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:21.304636 3002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:21.305832 3002 log.cc:824] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:21.309212 3002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: No bootstrap required, opened a new log
I20260928 16:44:21.312260 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.290105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.26.101.62:46567)
0928 16:44:21.290912 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.290926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.290976 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:44:21.292640 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.292667 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.293073 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.293409 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.295378 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.296880 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.298666 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.299448 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.299591 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.304402 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.304411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.304427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.304473 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.307984 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.308873 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.308879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.308886 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.309059 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.309492 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.309496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.309497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.310036 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.310254 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.310733 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.311149 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":587,"thread_start_us":372,"threads_started":1}
I20260928 16:44:21.317721 3002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:21.318132 3002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:21.318234 3002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e108258c84499593b751de17f24653, State: Initialized, Role: FOLLOWER
I20260928 16:44:21.318809 3002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:21.319056 3002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:21.319211 3002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:21.319375 3002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:21.321624 3002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:21.322158 3002 leader_election.cc:302] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e108258c84499593b751de17f24653; no voters:
I20260928 16:44:21.322727 3002 leader_election.cc:288] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:21.322860 3007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:21.323519 3007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 LEADER]: Becoming Leader. State: Replica: 25e108258c84499593b751de17f24653, State: Running, Role: LEADER
I20260928 16:44:21.324065 3007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:21.324781 3002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:21.327087 3008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e108258c84499593b751de17f24653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } } }
I20260928 16:44:21.327768 3008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:21.327119 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e108258c84499593b751de17f24653. Latest consensus state: current_term: 1 leader_uuid: "25e108258c84499593b751de17f24653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } } }
I20260928 16:44:21.331610 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:21.331670 3014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:21.338513 3014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:21.348400 3014 catalog_manager.cc:1347] Generated new cluster ID: 494113bedc654edea9c67996df5da073
I20260928 16:44:21.348536 3014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:21.381484 3014 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:21.383517 3014 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:21.396667 3014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Generated new TSK 0
I20260928 16:44:21.397594 3014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:44:21.466816 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:21.795845 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:21.796177 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:21.796295 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:21.803835 3028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:21.803961 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:21.804082 3028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:21.814005 3028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:21.815306 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:21.816629 3028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:21.829291 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.829347 3033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:21.829551 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:21.830363 3028 server_base.cc:1034] running on GCE node
I20260928 16:44:21.831003 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:21.832080 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:21.833284 3028 hybrid_clock.cc:648] HybridClock initialized: now 1790613861833228 us; error 43 us; skew 500 ppm
I20260928 16:44:21.833609 3028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:21 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:21.836802 3028 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:44:21.838282 3028 webserver.cc:460] Webserver started at http://127.26.101.1:34821/ using document root <none> and password file <none>
I20260928 16:44:21.838982 3028 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:21.839120 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:21.839416 3028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:21.842338 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "7998d14d20114bb9b8656e669cc66c99"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f7823de9e538412fc5098c7a3c4ea52b"
server_key_iv: "eb1b1f733d925e695fb08c6627c49b9c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.843142 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "7998d14d20114bb9b8656e669cc66c99"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f7823de9e538412fc5098c7a3c4ea52b"
server_key_iv: "eb1b1f733d925e695fb08c6627c49b9c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.848169 3028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:44:21.851531 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.852849 3028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 16:44:21.853018 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "7998d14d20114bb9b8656e669cc66c99"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f7823de9e538412fc5098c7a3c4ea52b"
server_key_iv: "eb1b1f733d925e695fb08c6627c49b9c"
server_key_version: "encryptionkey@0"
I20260928 16:44:21.853206 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:21.867266 3028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:21.871217 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:21.871498 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:21.872442 3028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:21.873793 3028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:21.873869 3028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.873981 3028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:21.874037 3028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:21.904156 3028 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:36077
I20260928 16:44:21.904199 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:36077 every 8 connection(s)
I20260928 16:44:21.905408 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:21.908409 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3028
I20260928 16:44:21.908612 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:21.909474 27028 external_mini_cluster.cc:1428] Setting key dda817c3cf126b05ef23a65016648f01
Sep 28 16:44:21 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613861, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:21.926966 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.907932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:32783 (local address 127.26.101.62:46567)
0928 16:44:21.908451 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:21.908458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:21.909511 (+ 1053us) server_negotiation.cc:407] Connection header received
0928 16:44:21.911734 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:21.911738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:21.911818 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:21.911909 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:21.913944 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.914665 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.916758 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.917164 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.917266 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.921186 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:21.921192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:21.921195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:21.921231 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:21.923763 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.924551 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.924555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.924557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.924683 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:21.925138 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:21.925142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:21.925144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:21.925425 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:21.925573 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:21.926336 (+ 763us) server_negotiation.cc:299] Negotiation successful
0928 16:44:21.926588 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":255,"threads_started":1}
I20260928 16:44:21.927982 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:21.908459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:32783)
0928 16:44:21.909306 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:21.909368 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:21.911429 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:21.912202 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:21.912215 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:21.912927 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:21.913762 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.913789 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.914834 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.914839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.916602 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:21.916618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:21.917390 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:21.917395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:21.917683 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:21.918406 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:21.918435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:21.920916 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:21.923929 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.923948 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.923965 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.924408 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.924792 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:21.924797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:21.924801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:21.925015 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:21.925567 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:21.925575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:21.925944 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 16:44:21.926186 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 16:44:21.926751 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":291,"threads_started":1}
I20260928 16:44:21.929795 3157 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46567
I20260928 16:44:21.930107 3157 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:21.930845 3157 heartbeater.cc:511] Master 127.26.101.62:46567 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:44:21.933185 2965 ts_manager.cc:194] Registered new tserver with Master: 7998d14d20114bb9b8656e669cc66c99 (127.26.101.1:36077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:44:21.934767 2965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:32783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:44:21.963317 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:22.295567 3163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:22.295864 3163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:22.295975 3163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:22.303526 3163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:22.303653 3163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:22.303768 3163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:22.313747 3163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:22.315044 3163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:22.316643 3163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:22.330282 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:22.330210 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:22.330250 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:22.331357 3163 server_base.cc:1034] running on GCE node
I20260928 16:44:22.332103 3163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:22.333199 3163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:22.334414 3163 hybrid_clock.cc:648] HybridClock initialized: now 1790613862334351 us; error 58 us; skew 500 ppm
I20260928 16:44:22.334748 3163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:22 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613862, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:22.338110 3163 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:44:22.339951 3163 webserver.cc:460] Webserver started at http://127.26.101.2:41667/ using document root <none> and password file <none>
I20260928 16:44:22.340705 3163 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:22.340835 3163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:22.341158 3163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:22.344728 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "92c752b3a7d74d4984e8b7e7a15ff7a7"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "baef7c8be0d3da48b6edcc3448771d58"
server_key_iv: "8601cce0a5a97f6a52033e534f3430f4"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.345587 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "92c752b3a7d74d4984e8b7e7a15ff7a7"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "baef7c8be0d3da48b6edcc3448771d58"
server_key_iv: "8601cce0a5a97f6a52033e534f3430f4"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.351373 3163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:44:22.355163 3178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.356676 3163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:22.356851 3163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "92c752b3a7d74d4984e8b7e7a15ff7a7"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "baef7c8be0d3da48b6edcc3448771d58"
server_key_iv: "8601cce0a5a97f6a52033e534f3430f4"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.357100 3163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:22.377326 3163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:22.381916 3163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:22.382290 3163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:22.383368 3163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:22.384761 3163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:22.384837 3163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.384938 3163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:22.384999 3163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.417099 3163 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:37931
I20260928 16:44:22.417155 3290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:37931 every 8 connection(s)
I20260928 16:44:22.418471 3163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:22.426090 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3163
I20260928 16:44:22.426278 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:22.426725 27028 external_mini_cluster.cc:1428] Setting key 90c556a1caf9f0629cc7e61e625d3772
Sep 28 16:44:22 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613862, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:22.439826 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:22.421111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58281 (local address 127.26.101.62:46567)
0928 16:44:22.421328 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:22.421333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:22.422489 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 16:44:22.424444 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:22.424451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:22.424536 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:22.424627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:22.426217 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.427014 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.428648 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.429172 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.429329 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:22.433028 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:22.433034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:22.433039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:22.433080 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:22.436241 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.437303 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.437307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.437310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.437463 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.437957 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.437962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.437965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.438325 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:22.438488 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:22.439240 (+ 752us) server_negotiation.cc:299] Negotiation successful
0928 16:44:22.439490 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:44:22.440843 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:22.421591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:58281)
0928 16:44:22.422317 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:22.422356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:22.424158 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:22.424882 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:22.424890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:22.425467 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:22.426038 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:22.426056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.427202 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.427206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:22.428446 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:22.428460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.429412 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.429418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:22.429634 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:22.430283 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:22.430313 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:22.432762 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:22.436439 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:22.436450 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:22.436472 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:22.437132 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:22.437586 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:22.437591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:22.437594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:22.437834 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:22.438444 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:22.438450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:22.438805 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 16:44:22.439130 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 16:44:22.439569 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":273,"threads_started":1}
I20260928 16:44:22.442452 3291 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46567
I20260928 16:44:22.442765 3291 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:22.443562 3291 heartbeater.cc:511] Master 127.26.101.62:46567 requested a full tablet report, sending...
I20260928 16:44:22.445250 2965 ts_manager.cc:194] Registered new tserver with Master: 92c752b3a7d74d4984e8b7e7a15ff7a7 (127.26.101.2:37931)
I20260928 16:44:22.446555 2965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:58281
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:44:22.479698 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:22.802263 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:22.802541 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:22.802663 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:22.809959 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:22.810071 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:22.810227 3296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:22.820247 3296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:22.821581 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:22.822886 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:22.836182 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:22.836158 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:22.836573 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:22.837092 3296 server_base.cc:1034] running on GCE node
I20260928 16:44:22.837769 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:22.838780 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:22.839982 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790613862839934 us; error 34 us; skew 500 ppm
I20260928 16:44:22.840323 3296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:22 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613862, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:22.843631 3296 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:44:22.845098 3296 webserver.cc:460] Webserver started at http://127.26.101.3:34287/ using document root <none> and password file <none>
I20260928 16:44:22.845803 3296 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:22.845927 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:22.846247 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:22.849243 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "710b843af84d4cad963bc981e2e8c3ad"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "5b973c3620ebf703ae182e8ee26e42b0"
server_key_iv: "0659e5b2abaecfda778920298c8cbb92"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.850032 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "710b843af84d4cad963bc981e2e8c3ad"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "5b973c3620ebf703ae182e8ee26e42b0"
server_key_iv: "0659e5b2abaecfda778920298c8cbb92"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.855376 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:44:22.858976 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.860452 3296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:22.860690 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "710b843af84d4cad963bc981e2e8c3ad"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "5b973c3620ebf703ae182e8ee26e42b0"
server_key_iv: "0659e5b2abaecfda778920298c8cbb92"
server_key_version: "encryptionkey@0"
I20260928 16:44:22.860888 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:22.877017 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:22.881233 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:22.881506 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:22.882423 3296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:22.883756 3296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:22.883836 3296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.883955 3296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:22.884014 3296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:22.913797 3296 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:34707
I20260928 16:44:22.913833 3425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:34707 every 8 connection(s)
I20260928 16:44:22.915011 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:22.920552 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3296
I20260928 16:44:22.920737 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:22.921243 27028 external_mini_cluster.cc:1428] Setting key 71bd161c0ac1dd29843204a4c844689a
Sep 28 16:44:22 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613862, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:22.935160 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:22.917638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:39875 (local address 127.26.101.62:46567)
0928 16:44:22.917820 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:22.917825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:22.918935 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 16:44:22.921060 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:22.921065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:22.921146 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:22.921249 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:22.923059 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.923946 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.925630 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.926072 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.926165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:22.929586 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:22.929591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:22.929595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:22.929630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:22.932192 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.932996 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.933000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.933002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.933136 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:22.933589 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:22.933592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:22.933594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:22.933873 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:22.933996 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:22.934620 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 16:44:22.934790 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:44:22.936343 3428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:22.918038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:39875)
0928 16:44:22.918748 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:22.918794 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:22.920726 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:22.921528 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:22.921539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:22.922141 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:22.922874 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:22.922902 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.924145 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.924151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:22.925491 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:22.925506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:22.926268 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:22.926273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:22.926494 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:22.927178 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:22.927201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:22.929350 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:22.932403 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:22.932410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:22.932424 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:22.932877 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:22.933260 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:22.933264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:22.933266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:22.933501 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:22.933978 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:22.933984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:22.934299 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 16:44:22.934543 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 16:44:22.935048 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260928 16:44:22.937773 3426 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46567
I20260928 16:44:22.938015 3157 heartbeater.cc:503] Master 127.26.101.62:46567 was elected leader, sending a full tablet report...
I20260928 16:44:22.938105 3426 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:22.938997 3426 heartbeater.cc:511] Master 127.26.101.62:46567 requested a full tablet report, sending...
I20260928 16:44:22.940501 2965 ts_manager.cc:194] Registered new tserver with Master: 710b843af84d4cad963bc981e2e8c3ad (127.26.101.3:34707)
I20260928 16:44:22.941282 2965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:39875
I20260928 16:44:22.949559 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
Entry for principal oryx/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab.
WARNING: no policy specified for oryx/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
Entry for principal oryx/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab.
WARNING: no policy specified for oryx/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
Entry for principal oryx/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab.
I20260928 16:44:23.089962 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3028
Sep 28 16:44:23 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:44:23.107434 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.091170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.26.101.1:36077)
0928 16:44:23.091777 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:23.091785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:23.091815 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:44:23.091961 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:23.091968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:23.092194 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:23.092339 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:23.093737 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.094820 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.096178 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.096767 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.096868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.101193 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:23.101210 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:23.101229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:23.101280 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:23.104452 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.105337 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.105345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.105348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.105527 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.105896 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.105900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.105903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.106275 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:23.106539 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:23.106792 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 16:44:23.107029 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":286,"threads_started":1}
I20260928 16:44:23.108747 3152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:23.192948 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3028
I20260928 16:44:23.208168 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3163
Sep 28 16:44:23 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:44:23.224817 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.209258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.26.101.2:37931)
0928 16:44:23.210034 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:23.210044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:23.210077 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:44:23.210165 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:23.210173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:23.210496 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:23.210696 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:23.212014 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.213135 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.214388 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.214944 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.215044 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.219252 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:23.219265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:23.219281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:23.219330 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:23.222048 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.222791 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.222798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.222801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.222964 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.223393 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.223397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.223399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.223761 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:23.224027 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:23.224287 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:44:23.224523 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":425,"threads_started":1}
I20260928 16:44:23.226166 3286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:23.309947 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3163
I20260928 16:44:23.324453 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3296
Sep 28 16:44:23 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:44:23.340934 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:23.325340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.26.101.3:34707)
0928 16:44:23.325977 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:23.325984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:23.326014 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:44:23.326076 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:23.326083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:23.326300 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:23.326432 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:23.327751 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.328807 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.330108 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:23.330683 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:23.330780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:23.335247 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:23.335261 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:23.335277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:23.335314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:23.338204 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.338938 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.338944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.338947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.339069 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:23.339498 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:23.339501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:23.339503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:23.339907 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:23.340182 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:23.340406 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 16:44:23.340629 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":302,"threads_started":1}
I20260928 16:44:23.342180 3421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:23.424649 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3296
I20260928 16:44:23.439483 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 2933
I20260928 16:44:23.440833 2996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:23.545636 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 2933
I20260928 16:44:23.564653 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46567
--webserver_interface=127.26.101.62
--webserver_port=36815
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:23.901409 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:23.901741 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:23.901870 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:23.909659 3443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:23.909793 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:23.909845 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:23.909924 3443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:23.909984 3443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:23.920446 3443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:23.921855 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:23.923362 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:23.935439 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:23.935487 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:23.935632 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:23.936578 3443 server_base.cc:1034] running on GCE node
I20260928 16:44:23.937426 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:23.938941 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:23.940145 3443 hybrid_clock.cc:648] HybridClock initialized: now 1790613863940083 us; error 60 us; skew 500 ppm
I20260928 16:44:23.940563 3443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:23 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613863, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:23.943858 3443 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:44:23.945585 3443 webserver.cc:460] Webserver started at http://127.26.101.62:36815/ using document root <none> and password file <none>
I20260928 16:44:23.946318 3443 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:23.946465 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:23.952389 3443 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:23.955590 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:23.957233 3443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:44:23.957428 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "25e108258c84499593b751de17f24653"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f6a37a885b756d0288bbb07f28ad2026"
server_key_iv: "40a7a5661257d5cd03173a87b5d11072"
server_key_version: "encryptionkey@0"
I20260928 16:44:23.958473 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:23.975931 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:23.980584 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:23.980870 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:23.999414 3443 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:46567
I20260928 16:44:23.999440 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:46567 every 8 connection(s)
I20260928 16:44:24.001093 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:24.007251 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3443
I20260928 16:44:24.008064 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:36077
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:46567
--webserver_port=34821
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:44:24.008756 3511 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:44:24.012218 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Bootstrap starting.
I20260928 16:44:24.031773 3511 log.cc:824] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:24.041538 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:44:24.042130 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Bootstrap complete.
I20260928 16:44:24.051573 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:24.052536 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e108258c84499593b751de17f24653, State: Initialized, Role: FOLLOWER
I20260928 16:44:24.053205 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:24.053467 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:24.053586 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:24.053752 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:44:24.057929 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:24.058573 3511 leader_election.cc:302] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e108258c84499593b751de17f24653; no voters:
I20260928 16:44:24.059307 3511 leader_election.cc:288] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:44:24.059458 3515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:44:24.060657 3515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [term 2 LEADER]: Becoming Leader. State: Replica: 25e108258c84499593b751de17f24653, State: Running, Role: LEADER
I20260928 16:44:24.061378 3515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } }
I20260928 16:44:24.061774 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:24.064065 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e108258c84499593b751de17f24653. Latest consensus state: current_term: 2 leader_uuid: "25e108258c84499593b751de17f24653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } } }
I20260928 16:44:24.064520 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:24.065069 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "25e108258c84499593b751de17f24653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e108258c84499593b751de17f24653" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46567 } } }
I20260928 16:44:24.065418 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:24.075681 3521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:24.082155 3521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:24.083956 3521 catalog_manager.cc:1259] Loaded cluster ID: 494113bedc654edea9c67996df5da073
I20260928 16:44:24.084172 3521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:24.088418 3521 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:24.090410 3521 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25e108258c84499593b751de17f24653: Loaded TSK: 0
I20260928 16:44:24.092355 3521 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:44:24.362840 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:24.363224 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:24.363404 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:24.370848 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:24.370980 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:24.371171 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:24.381947 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.1.keytab
--principal=oryx/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:36077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=34821
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:24.383476 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:24.384871 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:24.399863 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.400117 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.400480 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:24.401024 3513 server_base.cc:1034] running on GCE node
I20260928 16:44:24.401674 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:24.402796 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:24.404032 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790613864403959 us; error 67 us; skew 500 ppm
I20260928 16:44:24.404354 3513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:24 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:24.407711 3513 init.cc:377] Logged in from keytab as oryx/127.26.101.1@KRBTEST.COM (short username oryx)
I20260928 16:44:24.409600 3513 webserver.cc:460] Webserver started at http://127.26.101.1:34821/ using document root <none> and password file <none>
I20260928 16:44:24.410383 3513 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:24.410530 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:24.416271 3513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:44:24.419209 3547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.420821 3513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:44:24.420993 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "7998d14d20114bb9b8656e669cc66c99"
format_stamp: "Formatted at 2026-09-28 16:44:21 on dist-test-slave-t0tw"
server_key: "f7823de9e538412fc5098c7a3c4ea52b"
server_key_iv: "eb1b1f733d925e695fb08c6627c49b9c"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.422062 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:24.433255 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:24.437485 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:24.437755 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:24.438771 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:24.440179 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:24.440294 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.440415 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:24.440485 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.470384 3513 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:36077
I20260928 16:44:24.470466 3660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:36077 every 8 connection(s)
I20260928 16:44:24.471940 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:24.481717 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3513
I20260928 16:44:24.482345 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:37931
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:46567
--webserver_port=41667
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:44:24 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:24 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:24.817099 3665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:24.817633 3665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:24.817818 3665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:24.831233 3665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:24.831434 3665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:24.831655 3665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:24.849854 3665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.2.keytab
--principal=oryx/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=41667
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:24.852070 3665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:24.854215 3665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:24.871973 3672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.872072 3671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:24.873193 3674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:24.873545 3665 server_base.cc:1034] running on GCE node
I20260928 16:44:24.874629 3665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:24.876134 3665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:24.877419 3665 hybrid_clock.cc:648] HybridClock initialized: now 1790613864877318 us; error 94 us; skew 500 ppm
I20260928 16:44:24.877939 3665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:24 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613864, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:24.882798 3665 init.cc:377] Logged in from keytab as oryx/127.26.101.2@KRBTEST.COM (short username oryx)
I20260928 16:44:24.885054 3665 webserver.cc:460] Webserver started at http://127.26.101.2:41667/ using document root <none> and password file <none>
I20260928 16:44:24.886194 3665 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:24.886389 3665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:24.895673 3665 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.007s
I20260928 16:44:24.899899 3681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.902086 3665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:44:24.902359 3665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "92c752b3a7d74d4984e8b7e7a15ff7a7"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "baef7c8be0d3da48b6edcc3448771d58"
server_key_iv: "8601cce0a5a97f6a52033e534f3430f4"
server_key_version: "encryptionkey@0"
I20260928 16:44:24.904103 3665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:24.963541 3665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:24.970355 3665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:24.970726 3665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:24.972179 3665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:24.974388 3665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:24.974534 3665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:24.974671 3665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:24.974751 3665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:25.016032 3665 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:37931
I20260928 16:44:25.016073 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:37931 every 8 connection(s)
I20260928 16:44:25.018337 3665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:25.025575 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3665
I20260928 16:44:25.026335 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:34707
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:46567
--webserver_port=34287
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:25.563374 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:25.563894 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:25.564079 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:25.577461 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:25.577664 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:25.577883 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:25.588953 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/krb5kdc/oryx_127.26.101.3.keytab
--principal=oryx/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:34707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=34287
--tserver_master_addrs=127.26.101.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:25.590360 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:25.591974 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:25.607580 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:25.608713 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:25.609655 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:25.611209 3799 server_base.cc:1034] running on GCE node
I20260928 16:44:25.612000 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:25.613114 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:25.614378 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790613865614314 us; error 59 us; skew 500 ppm
I20260928 16:44:25.614722 3799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613865, etypes {rep=17 tkt=17 ses=17}, oryx/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:25.618060 3799 init.cc:377] Logged in from keytab as oryx/127.26.101.3@KRBTEST.COM (short username oryx)
I20260928 16:44:25.619854 3799 webserver.cc:460] Webserver started at http://127.26.101.3:34287/ using document root <none> and password file <none>
I20260928 16:44:25.620622 3799 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:25.620771 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:25.626565 3799 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:44:25.629503 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:25.630972 3799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:25.631160 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "710b843af84d4cad963bc981e2e8c3ad"
format_stamp: "Formatted at 2026-09-28 16:44:22 on dist-test-slave-t0tw"
server_key: "5b973c3620ebf703ae182e8ee26e42b0"
server_key_iv: "0659e5b2abaecfda778920298c8cbb92"
server_key_version: "encryptionkey@0"
I20260928 16:44:25.632148 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:25.653010 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:25.657622 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:25.657914 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:25.659052 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:25.660666 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:25.660768 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:25.660892 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:25.660972 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:25.707058 3799 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:34707
I20260928 16:44:25.707110 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:34707 every 8 connection(s)
I20260928 16:44:25.708734 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:25.711776 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 3799
W20260928 16:44:25.725824 3664 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.240s user 0.000s sys 0.012s
I20260928 16:44:25.728475 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:24.475015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:51499)
0928 16:44:24.475885 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:24.475923 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:24.477686 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:24.479190 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:24.479200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:24.479803 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:24.480371 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.480389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.482001 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.482005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.483331 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:24.483347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.484399 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.484408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:24.484662 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:24.485290 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:24.485312 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:25.727011 (+1241699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":326,"threads_started":1}
W20260928 16:44:25.730125 3661 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:46567 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.101.62:46567: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
W20260928 16:44:25.731139 3663 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:24.474990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:51499 (local address 127.26.101.62:46567)
0928 16:44:24.475790 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:24.475805 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:24.476053 (+ 248us) server_negotiation.cc:407] Connection header received
0928 16:44:24.478254 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:24.478268 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:24.478627 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:24.478960 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:24.480561 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.481813 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.483475 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:24.484158 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:24.484327 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.729942 (+1245615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:51499: BlockingRecv error: recv got EOF from 127.26.101.1:51499 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":268,"threads_started":1}
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:25.750294 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:25.731032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:51467)
0928 16:44:25.732106 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:25.732130 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:25.732276 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:25.734249 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:25.734255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:25.734823 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:25.735118 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.735134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.739782 (+ 4648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.739790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.741603 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.741624 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.743960 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.743968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.744198 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.744842 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:25.744869 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:25.749913 (+ 5044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":913}
W20260928 16:44:25.751562 3936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:25.731637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:51467 (local address 127.26.101.62:46567)
0928 16:44:25.733919 (+ 2282us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:25.733924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:25.733940 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:25.733991 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:25.734000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:25.734063 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:25.735327 (+ 1264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:25.735615 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.736954 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.741863 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.742572 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.744650 (+ 2078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.751001 (+ 6351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:51467: BlockingRecv error: recv got EOF from 127.26.101.1:51467 (error 108)
Metrics: {"server-negotiator.queue_time_us":2153}
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:25 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:25.766359 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:25.713777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.26.101.62:46567)
0928 16:44:25.732923 (+ 19146us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:25.732930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:25.732952 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:25.733038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:25.733044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:25.733136 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:25.733271 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:25.738052 (+ 4781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.739533 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.742715 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.743424 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.743573 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.748106 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:25.748120 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:25.748149 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:25.748208 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:25.761505 (+ 13297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:25.762515 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:25.762528 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:25.762540 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:25.762874 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:25.763446 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:25.763452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:25.763456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:25.764349 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:25.764760 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:25.765337 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 16:44:25.765849 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19021,"thread_start_us":19903,"threads_started":1}
I20260928 16:44:25.767534 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:25.751878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:47467)
0928 16:44:25.752218 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:25.752241 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:25.752373 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:25.753456 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:25.753462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:25.753950 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:25.754215 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.754230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.756653 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.756661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.758463 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.758488 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.759660 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.759667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.759903 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.760547 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:25.760572 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:25.767258 (+ 6686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205}
W20260928 16:44:25.767721 3936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:25.751522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:47467 (local address 127.26.101.62:46567)
0928 16:44:25.753044 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:25.753052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:25.753075 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:44:25.753142 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:25.753148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:25.753237 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:25.754403 (+ 1166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:25.755016 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.756421 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.758690 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.759439 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.760256 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.767473 (+ 7217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:47467: BlockingRecv error: recv got EOF from 127.26.101.1:47467 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":3083,"threads_started":1}
W20260928 16:44:25.769713 3661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:44:26.060261 3798 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.027s user 0.000s sys 0.012s
W20260928 16:44:26.061851 3797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:25.021259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:59759 (local address 127.26.101.62:46567)
0928 16:44:25.021835 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:25.021844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:25.022752 (+ 908us) server_negotiation.cc:407] Connection header received
0928 16:44:25.024668 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:25.024675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:25.024790 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:25.024944 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:25.027106 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.028501 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.030641 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.031403 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.031553 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.061420 (+1029867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:59759: BlockingRecv error: recv got EOF from 127.26.101.2:59759 (error 108)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260928 16:44:26.061946 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:25.021739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:59759)
0928 16:44:25.022581 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:25.022628 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:25.024399 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:25.025162 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:25.025173 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:25.026017 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:25.026832 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.026858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.028706 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.028712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.030480 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.030496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.031672 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.031678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.031984 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.032867 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:25.032902 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.061022 (+1028120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":308,"threads_started":1}
W20260928 16:44:26.062855 3795 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:46567 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.101.62:46567: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:26.073271 3797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:26.063663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58347 (local address 127.26.101.62:46567)
0928 16:44:26.063826 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.063832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.063852 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:26.063993 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.063999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.064067 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.064149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.064900 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.065663 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.066925 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.067409 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.067521 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.073094 (+ 5573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:58347: BlockingRecv error: recv got EOF from 127.26.101.2:58347 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 16:44:26.073275 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.063540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:58347)
0928 16:44:26.063715 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.063738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.063870 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.064199 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.064203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.064582 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.064755 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.064766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.065818 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.065822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.066782 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.066794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.067584 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.067592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.067750 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.068344 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.068362 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.072884 (+ 4522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:26.083806 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.074200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:34045)
0928 16:44:26.074364 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.074380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.074495 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.074951 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.074956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.075324 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.075487 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.075496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.076602 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.076606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.077676 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.077690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.078444 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.078450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.078630 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.079344 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.079367 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.083465 (+ 4098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 16:44:26.083875 3797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:26.074478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:34045 (local address 127.26.101.62:46567)
0928 16:44:26.074662 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.074666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.074680 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:26.074728 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.074731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.074806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.074891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.075628 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.076459 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.077810 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.078297 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.078406 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.083657 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:34045: BlockingRecv error: recv got EOF from 127.26.101.2:34045 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:44:26.084201 3795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:44:26.746906 3934 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.995s user 0.002s sys 0.013s
W20260928 16:44:26.748227 3663 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:25.713510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35169 (local address 127.26.101.62:46567)
0928 16:44:25.731732 (+ 18222us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:25.731740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:25.738684 (+ 6944us) server_negotiation.cc:407] Connection header received
0928 16:44:25.741746 (+ 3062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:25.741752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:25.741848 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:25.741919 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:25.744698 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.745843 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.748999 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.749647 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.750086 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.747950 (+997864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:35169: BlockingRecv error: recv got EOF from 127.26.101.3:35169 (error 108)
Metrics: {"server-negotiator.queue_time_us":18080,"thread_start_us":19133,"threads_started":1}
I20260928 16:44:26.748942 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:25.712794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:35169)
0928 16:44:25.737908 (+ 25114us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:25.738558 (+ 650us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:25.741286 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:25.742598 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:25.742612 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:25.743575 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:25.744474 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.744509 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.746092 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.746099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.748834 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:25.748850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:25.750423 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:25.750432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:25.750744 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:25.751571 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:25.751609 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.747752 (+996143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24668,"thread_start_us":24367,"threads_started":1}
W20260928 16:44:26.749785 3931 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:46567 (0 consecutive failures): Not authorized: Failed to ping master at 127.26.101.62:46567: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:26.764978 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.750532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:49489)
0928 16:44:26.750709 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.750727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.750830 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.751175 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.751178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.751563 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.751731 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.751740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.752673 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.752677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.753682 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.753694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.754437 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.754442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.754561 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.760895 (+ 6334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.760916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.764729 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:44:26.765228 3663 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:26.750570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:49489 (local address 127.26.101.62:46567)
0928 16:44:26.750713 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.750717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.750783 (+ 66us) server_negotiation.cc:407] Connection header received
0928 16:44:26.750948 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.750953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.751031 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.751160 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.751840 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.752535 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.753840 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.754288 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.754402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.764998 (+ 10596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:49489: BlockingRecv error: recv got EOF from 127.26.101.3:49489 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:44:26.775520 3663 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:26.765798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36549 (local address 127.26.101.62:46567)
0928 16:44:26.766057 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.766063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.766080 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:26.766130 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.766134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.766193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.766274 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.766980 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.767725 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.768998 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.769429 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.769539 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.775240 (+ 5701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:36549: BlockingRecv error: recv got EOF from 127.26.101.3:36549 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 16:44:26.776057 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.765703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:36549)
0928 16:44:26.765844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.765860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.765982 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.766329 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.766334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.766658 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.766822 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.766832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.767882 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.767886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.768843 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.768855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.769579 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.769585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.769763 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.771110 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.771134 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.775856 (+ 4722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:44:26.776386 3931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:26 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:26.779938 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:26.770264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:44161)
0928 16:44:26.770803 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:26.770820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:26.770921 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:26.771431 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:26.771434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:26.771814 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:26.771958 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.771967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.773031 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.773037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.774589 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:26.774607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.775662 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.775668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:26.775838 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.776434 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:26.776452 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:26.779692 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
W20260928 16:44:26.780145 3940 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:26.770337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:44161 (local address 127.26.101.62:46567)
0928 16:44:26.771139 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:26.771145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:26.771163 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:26.771210 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:26.771215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:26.771297 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:26.771407 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:26.772103 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.772872 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.774781 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:26.775450 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:26.775595 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:26.779909 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:44161: BlockingRecv error: recv got EOF from 127.26.101.1:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":537,"threads_started":1}
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:27.093772 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.084756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:49825)
0928 16:44:27.085269 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.085287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.085433 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.085772 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.085775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.086182 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:27.086329 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.086338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.087450 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.087454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.088422 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.088436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.089199 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.089204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.089330 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.089963 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:27.089981 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:27.093528 (+ 3547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":278,"threads_started":1}
W20260928 16:44:27.094086 3940 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:27.084839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:49825 (local address 127.26.101.62:46567)
0928 16:44:27.085091 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.085097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.085383 (+ 286us) server_negotiation.cc:407] Connection header received
0928 16:44:27.085576 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.085581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.085651 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.085744 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.086516 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.087309 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.088574 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.089021 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.089140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.093856 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:49825: BlockingRecv error: recv got EOF from 127.26.101.2:49825 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:27.788002 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.777178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:45819)
0928 16:44:27.777915 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.777933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.778055 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.778473 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.778476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.778969 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:27.779201 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.779213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.780355 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.780360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.781783 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.781803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.782848 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.782853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.782984 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.783685 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:27.783710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:27.787730 (+ 4020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":327,"threads_started":1}
W20260928 16:44:27.788316 3944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:27.777337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:45819 (local address 127.26.101.62:46567)
0928 16:44:27.778032 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.778039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.778060 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:27.778170 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.778173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.778298 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.778410 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.779360 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.780153 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.781971 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.782635 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.782775 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.788021 (+ 5246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:45819: BlockingRecv error: recv got EOF from 127.26.101.3:45819 (error 108)
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":422,"threads_started":1}
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:27 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:27.791477 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:27.780906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:53707)
0928 16:44:27.781484 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:27.781508 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:27.781634 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:27.782139 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:27.782144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:27.782704 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:27.782951 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.782967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.784417 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.784421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.785589 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:27.785604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.786503 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.786510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:27.786691 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.787360 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:27.787386 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:27.791110 (+ 3724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
W20260928 16:44:27.791682 3946 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:27.780975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53707 (local address 127.26.101.62:46567)
0928 16:44:27.781764 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:27.781769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:27.781799 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:44:27.781851 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:27.781855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:27.781956 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:27.782038 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:27.783150 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.784223 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.785736 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:27.786294 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:27.786418 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:27.791393 (+ 4975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:53707: BlockingRecv error: recv got EOF from 127.26.101.1:53707 (error 108)
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":541,"threads_started":1}
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:28.105593 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:28.094859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:39641)
0928 16:44:28.095575 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:28.095592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:28.095747 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:28.096194 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:28.096197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:28.096740 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:28.096958 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.096970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.098154 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.098160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.099248 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.099266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.100370 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.100377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.100538 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.101241 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:28.101262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:28.105190 (+ 3928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":481,"threads_started":1}
W20260928 16:44:28.105716 3946 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:28.094910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:39641 (local address 127.26.101.62:46567)
0928 16:44:28.095176 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:28.095184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:28.095660 (+ 476us) server_negotiation.cc:407] Connection header received
0928 16:44:28.095926 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:28.095931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:28.096000 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:28.096094 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:28.097198 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.097970 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.099530 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.100052 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.100206 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.105461 (+ 5255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:39641: BlockingRecv error: recv got EOF from 127.26.101.2:39641 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:28.799645 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:28.789024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:47635)
0928 16:44:28.789573 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:28.789591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:28.789716 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:28.790265 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:28.790269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:28.790667 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:28.790860 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.790869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.791990 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.791994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.793275 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.793292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.794212 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.794218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.794388 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.795114 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:28.795138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:28.799287 (+ 4149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":322,"threads_started":1}
W20260928 16:44:28.799741 3949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:28.789224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:47635 (local address 127.26.101.62:46567)
0928 16:44:28.789905 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:28.789910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:28.789926 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:28.789974 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:28.789977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:28.790046 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:28.790136 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:28.791023 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.791839 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.793412 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.794042 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.794132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.799501 (+ 5369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:47635: BlockingRecv error: recv got EOF from 127.26.101.3:47635 (error 108)
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":445,"threads_started":1}
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:28 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:28.802320 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:28.792558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:50003)
0928 16:44:28.793371 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:28.793390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:28.793503 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:28.793802 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:28.793805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:28.794173 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:28.794378 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.794391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.795619 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.795624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.797020 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:28.797038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.797732 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.797737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:28.797885 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.798474 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:28.798497 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:28.802068 (+ 3571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":569,"threads_started":1}
W20260928 16:44:28.802521 3951 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:28.792682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:50003 (local address 127.26.101.62:46567)
0928 16:44:28.793224 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:28.793229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:28.793473 (+ 244us) server_negotiation.cc:407] Connection header received
0928 16:44:28.793592 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:28.793597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:28.793667 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:28.793771 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:28.794531 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.795437 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.797175 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:28.797581 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:28.797700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:28.802288 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:50003: BlockingRecv error: recv got EOF from 127.26.101.1:50003 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":249,"threads_started":1}
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:29.116446 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.106803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:46581)
0928 16:44:29.107399 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.107420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.107587 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.107960 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.107965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.108449 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.108645 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.108663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.109722 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.109726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.110695 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.110710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.111518 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.111523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.111693 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.112433 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.112453 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.116168 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
W20260928 16:44:29.116751 3951 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:29.106875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:46581 (local address 127.26.101.62:46567)
0928 16:44:29.107113 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.107119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.107540 (+ 421us) server_negotiation.cc:407] Connection header received
0928 16:44:29.107721 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.107727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.107804 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.107888 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.108793 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.109588 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.110852 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.111316 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.111439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.116475 (+ 5036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:46581: BlockingRecv error: recv got EOF from 127.26.101.2:46581 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:29.810686 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.800710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:35927)
0928 16:44:29.801304 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.801322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.801444 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.801929 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.801933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.802312 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.802493 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.802507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.803957 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.803963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.805224 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.805244 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.806046 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.806052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.806229 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.806802 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.806820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.810343 (+ 3523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":304,"threads_started":1}
W20260928 16:44:29.810850 3954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:29.800920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35927 (local address 127.26.101.62:46567)
0928 16:44:29.801639 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.801644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.801661 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:29.801715 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.801719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.801777 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.801876 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.802653 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.803735 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.805419 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.805892 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.806009 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.810590 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:35927: BlockingRecv error: recv got EOF from 127.26.101.3:35927 (error 108)
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":482,"threads_started":1}
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:29 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:29.813479 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:29.803458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:32839)
0928 16:44:29.804909 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:29.804930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:29.805049 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:29.805378 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:29.805383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:29.805781 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:29.805941 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.805951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.806977 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.806983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.808008 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:29.808022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.808767 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.808773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:29.808945 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.809597 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:29.809621 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:29.813131 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":961,"threads_started":1}
W20260928 16:44:29.813684 3956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:29.803504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:32839 (local address 127.26.101.62:46567)
0928 16:44:29.804960 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:29.804966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:29.804984 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:29.805153 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:29.805158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:29.805232 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:29.805317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:29.806097 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.806845 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.808144 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:29.808619 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:29.808724 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:29.813400 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:32839: BlockingRecv error: recv got EOF from 127.26.101.1:32839 (error 108)
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":1185,"threads_started":1}
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:30.127233 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.117538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:57781)
0928 16:44:30.118109 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.118127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.118274 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.118633 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.118638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.119107 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:30.119287 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.119301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.120430 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.120434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.121448 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.121462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.122198 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.122204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.122385 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.122992 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:30.123009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:30.126864 (+ 3855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":333,"threads_started":1}
W20260928 16:44:30.127439 3956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:30.117615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:57781 (local address 127.26.101.62:46567)
0928 16:44:30.117862 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.117868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.118222 (+ 354us) server_negotiation.cc:407] Connection header received
0928 16:44:30.118408 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.118412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.118481 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.118565 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:30.119450 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.120287 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.121603 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.122038 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.122154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.127143 (+ 4989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:57781: BlockingRecv error: recv got EOF from 127.26.101.2:57781 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:30.822312 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.811915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:48555)
0928 16:44:30.812477 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.812495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.812634 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.813042 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.813047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.813523 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:30.813749 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.813766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.815029 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.815045 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.816306 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.816322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.817185 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.817192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.817411 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.818098 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:30.818116 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:30.821939 (+ 3823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":317,"threads_started":1}
W20260928 16:44:30.822368 3959 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:30.812055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:48555 (local address 127.26.101.62:46567)
0928 16:44:30.812560 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.812565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.812587 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:30.812803 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.812809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.812879 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.812964 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:30.813925 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.814842 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.816452 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.817013 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.817103 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.822157 (+ 5054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:48555: BlockingRecv error: recv got EOF from 127.26.101.3:48555 (error 108)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
Sep 28 16:44:30 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:30.824424 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:30.814524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:55343)
0928 16:44:30.815429 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:30.815446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:30.815568 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:30.815878 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:30.815883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:30.816297 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:30.816483 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.816497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.817765 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.817771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.818779 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:30.818793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.819623 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.819630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:30.819844 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.820472 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:30.820490 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:30.824141 (+ 3651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":699,"threads_started":1}
W20260928 16:44:30.824656 3961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:30.814630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55343 (local address 127.26.101.62:46567)
0928 16:44:30.815161 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:30.815167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:30.815519 (+ 352us) server_negotiation.cc:407] Connection header received
0928 16:44:30.815672 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:30.815678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:30.815741 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:30.815828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:30.816630 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.817633 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.818931 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:30.819459 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:30.819570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:30.824376 (+ 4806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:55343: BlockingRecv error: recv got EOF from 127.26.101.1:55343 (error 108)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":243,"threads_started":1}
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:31.137413 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:31.128323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:40893)
0928 16:44:31.128798 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:31.128815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:31.128929 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:31.129215 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:31.129218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:31.129669 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:31.129828 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.129838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.130904 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.130909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.131915 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.131928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.132619 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.132623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.132752 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.133329 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:31.133372 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:31.137027 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
W20260928 16:44:31.137617 3961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:31.128490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:40893 (local address 127.26.101.62:46567)
0928 16:44:31.128729 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:31.128736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:31.128896 (+ 160us) server_negotiation.cc:407] Connection header received
0928 16:44:31.129036 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:31.129040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:31.129105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:31.129186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:31.129965 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.130756 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.132069 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.132498 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.132597 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.137363 (+ 4766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:40893: BlockingRecv error: recv got EOF from 127.26.101.2:40893 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:31 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:31.840411 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:31.823337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:43049)
0928 16:44:31.823843 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:31.823861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:31.823978 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:31.824344 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:31.824348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:31.824814 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:31.824972 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.824982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.826476 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.826482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.827874 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.827890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.828812 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.828818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.829031 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.836075 (+ 7044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:31.836097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:31.840101 (+ 4004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20260928 16:44:31.840708 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:31.825368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:33517)
0928 16:44:31.825865 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:31.825888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:31.826014 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:31.826581 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:31.826588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:31.827148 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:31.827383 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.827397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.828826 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.828831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.829965 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:31.829981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.830727 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.830731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:31.830860 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.836136 (+ 5276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:31.836165 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:31.840377 (+ 4212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
W20260928 16:44:31.840698 3964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:31.823343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:43049 (local address 127.26.101.62:46567)
0928 16:44:31.823827 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:31.823832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:31.823925 (+ 93us) server_negotiation.cc:407] Connection header received
0928 16:44:31.824084 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:31.824088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:31.824164 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:31.824262 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:31.825128 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.826343 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.828031 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.828665 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.828801 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.840415 (+ 11614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:43049: BlockingRecv error: recv got EOF from 127.26.101.3:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
W20260928 16:44:31.841194 3966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:31.825443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:33517 (local address 127.26.101.62:46567)
0928 16:44:31.826191 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:31.826196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:31.826214 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:44:31.826260 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:31.826265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:31.826332 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:31.826442 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:31.827565 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.828675 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.830115 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:31.830593 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:31.830689 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:31.840658 (+ 9969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:33517: BlockingRecv error: recv got EOF from 127.26.101.1:33517 (error 108)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":486,"threads_started":1}
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:32.148486 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.138737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:36239)
0928 16:44:32.139453 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:32.139470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:32.139590 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:32.139941 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:32.139945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:32.140357 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:32.140535 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.140545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.141603 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.141607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.142529 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.142542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.143422 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.143428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.143610 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.144365 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:32.144384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:32.148165 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":265,"threads_started":1}
W20260928 16:44:32.148610 3966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:32.138853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36239 (local address 127.26.101.62:46567)
0928 16:44:32.139129 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.139135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.139550 (+ 415us) server_negotiation.cc:407] Connection header received
0928 16:44:32.139689 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.139693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.139758 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.139845 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:32.140654 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.141464 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.142686 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.143251 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.143392 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.148389 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:36239: BlockingRecv error: recv got EOF from 127.26.101.2:36239 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:32 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:32.851920 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.841667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:36043)
0928 16:44:32.842139 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:32.842162 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:32.842299 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:32.843104 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:32.843110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:32.843539 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:32.843718 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.843728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.844769 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.844773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.845705 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.845718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.846393 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.846399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.846570 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.847325 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:32.847348 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:32.851587 (+ 4239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
W20260928 16:44:32.852137 3971 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:32.842354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36043 (local address 127.26.101.62:46567)
0928 16:44:32.842777 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.842781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.842793 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:44:32.842842 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.842845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.842917 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.843012 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:32.843864 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.844623 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.845844 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.846240 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.846350 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.851853 (+ 5503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:36043: BlockingRecv error: recv got EOF from 127.26.101.3:36043 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
W20260928 16:44:32.852320 3970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:32.841755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:34727 (local address 127.26.101.62:46567)
0928 16:44:32.842489 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:32.842495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:32.843120 (+ 625us) server_negotiation.cc:407] Connection header received
0928 16:44:32.843325 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:32.843328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:32.843384 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:32.843481 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:32.844321 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.845116 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.846354 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.846812 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.846933 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.851873 (+ 4940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:34727: BlockingRecv error: recv got EOF from 127.26.101.1:34727 (error 108)
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":470,"threads_started":1}
I20260928 16:44:32.852743 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:32.841623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:34727)
0928 16:44:32.843015 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:32.843033 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:32.843212 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:32.843546 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:32.843552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:32.843995 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:32.844162 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.844171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.845262 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.845268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.846213 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:32.846225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:32.846984 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:32.846989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:32.847194 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:32.847763 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:32.847786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:32.852527 (+ 4741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1281,"thread_start_us":1039,"threads_started":1}
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:33.159337 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:33.149580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:50485)
0928 16:44:33.150385 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:33.150402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:33.150536 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:33.150854 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:33.150857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:33.151330 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:33.151509 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.151519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.152801 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.152805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.153873 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.153886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.154634 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.154638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.154772 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.155449 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:33.155469 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:33.159051 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":420,"threads_started":1}
W20260928 16:44:33.159495 3970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:33.149702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:50485 (local address 127.26.101.62:46567)
0928 16:44:33.150006 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:33.150013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:33.150519 (+ 506us) server_negotiation.cc:407] Connection header received
0928 16:44:33.150653 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:33.150657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:33.150726 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:33.150820 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:33.151682 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.152640 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.153986 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.154494 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.154594 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.159286 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:50485: BlockingRecv error: recv got EOF from 127.26.101.2:50485 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:33 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:33.874364 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:33.853630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:45395)
0928 16:44:33.864031 (+ 10401us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:33.864057 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:33.864231 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:33.864869 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:33.864875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:33.865336 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:33.865513 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.865523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.866718 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.866722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.867695 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.867708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.868526 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.868533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.868701 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.869375 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:33.869393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:33.873962 (+ 4569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10201,"thread_start_us":10027,"threads_started":1}
I20260928 16:44:33.874368 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:33.853279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:44435)
0928 16:44:33.864324 (+ 11045us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:33.864345 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:33.864518 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:33.864881 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:33.864886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:33.865335 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:33.865520 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.865534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.866667 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.866673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.867667 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:33.867679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.868472 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.868484 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:33.868649 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.869232 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:33.869252 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:33.873970 (+ 4718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10919,"thread_start_us":10973,"threads_started":1}
W20260928 16:44:33.874572 3974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:33.854101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45395 (local address 127.26.101.62:46567)
0928 16:44:33.864507 (+ 10406us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:33.864513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:33.864530 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:33.864609 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:33.864614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:33.864688 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:33.864787 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:33.865667 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.866563 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.867841 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.868325 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.868442 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.874239 (+ 5797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:45395: BlockingRecv error: recv got EOF from 127.26.101.1:45395 (error 108)
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":10282,"spinlock_wait_cycles":13056,"thread_start_us":9886,"threads_started":1}
W20260928 16:44:33.874791 3976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:33.853417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:44435 (local address 127.26.101.62:46567)
0928 16:44:33.864332 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:33.864342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:33.864472 (+ 130us) server_negotiation.cc:407] Connection header received
0928 16:44:33.864664 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:33.864668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:33.864724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:33.864814 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:33.865674 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.866524 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.867821 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:33.868314 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:33.868423 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:33.874211 (+ 5788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:44435: BlockingRecv error: recv got EOF from 127.26.101.3:44435 (error 108)
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":10753,"spinlock_wait_cycles":4736,"thread_start_us":10930,"threads_started":1}
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:34.172556 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:34.160486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:59477)
0928 16:44:34.163529 (+ 3043us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:34.163548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:34.163690 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:34.163960 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:34.163964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:34.164364 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:34.164532 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.164542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.165834 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.165839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.166800 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.166814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.167588 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.167594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.167745 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.168396 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:34.168415 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:34.172254 (+ 3839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2903,"thread_start_us":2778,"threads_started":1}
W20260928 16:44:34.172842 3974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:34.160548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:59477 (local address 127.26.101.62:46567)
0928 16:44:34.160750 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:34.160756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:34.163637 (+ 2881us) server_negotiation.cc:407] Connection header received
0928 16:44:34.163782 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:34.163785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:34.163852 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:34.163935 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:34.164706 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.165671 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.166917 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.167401 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.167494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.172497 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:59477: BlockingRecv error: recv got EOF from 127.26.101.2:59477 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:34 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:34.886584 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:34.875477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:34537)
0928 16:44:34.876223 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:34.876250 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:34.876455 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:34.877134 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:34.877138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:34.877716 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:34.877913 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.877924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.879200 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.879206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.880354 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.880368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.881332 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.881340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.881503 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.882221 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:34.882240 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:34.886237 (+ 3997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":351,"threads_started":1}
I20260928 16:44:34.886660 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:34.875775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:57429)
0928 16:44:34.877129 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:34.877146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:34.877303 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:34.877681 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:34.877685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:34.878140 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:34.878334 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.878348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.879739 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.879743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.880910 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:34.880927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.881833 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.881839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:34.882014 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.882532 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:34.882549 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:34.886333 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":1104,"threads_started":1}
W20260928 16:44:34.886754 3979 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:34.875622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:34537 (local address 127.26.101.62:46567)
0928 16:44:34.876519 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:34.876527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:34.876549 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:34.876635 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:34.876641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:34.876743 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:34.876876 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:34.878089 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.879013 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.880512 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.881133 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.881232 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.886483 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:34537: BlockingRecv error: recv got EOF from 127.26.101.3:34537 (error 108)
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":1065,"threads_started":1}
W20260928 16:44:34.887025 3981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:34.875743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57429 (local address 127.26.101.62:46567)
0928 16:44:34.877014 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:34.877021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:34.877267 (+ 246us) server_negotiation.cc:407] Connection header received
0928 16:44:34.877392 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:34.877397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:34.877491 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:34.877600 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:34.878538 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.879566 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.881057 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:34.881659 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:34.881776 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:34.886575 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:57429: BlockingRecv error: recv got EOF from 127.26.101.1:57429 (error 108)
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":459,"threads_started":1}
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:35.182862 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:35.173773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:58609)
0928 16:44:35.174347 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:35.174364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:35.174512 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:35.174834 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:35.174839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:35.175284 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:35.175435 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.175445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.176506 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.176510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.177536 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.177551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.178263 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.178269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.178433 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.179032 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:35.179051 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:35.182508 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":303,"threads_started":1}
W20260928 16:44:35.183020 3981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:35.173868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58609 (local address 127.26.101.62:46567)
0928 16:44:35.174096 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:35.174103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:35.174462 (+ 359us) server_negotiation.cc:407] Connection header received
0928 16:44:35.174632 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:35.174636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:35.174700 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:35.174785 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:35.175592 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.176366 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.177678 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.178100 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.178221 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.182771 (+ 4550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:58609: BlockingRecv error: recv got EOF from 127.26.101.2:58609 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:35 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:35.898772 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:35.887891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:58771)
0928 16:44:35.889082 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:35.889104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:35.889467 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:35.890057 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:35.890061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:35.890584 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:35.890770 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.890781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.892008 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.892012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.892957 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.892971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.893735 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.893923 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.894561 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:35.894578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:35.898433 (+ 3855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":951,"threads_started":1}
I20260928 16:44:35.898810 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:35.887880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:47135)
0928 16:44:35.889489 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:35.889507 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:35.889653 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:35.890054 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:35.890058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:35.890563 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:35.890772 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.890786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.891951 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.891957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.892968 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:35.892982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.893796 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:35.893953 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.894627 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:35.894645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:35.898485 (+ 3840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1489,"thread_start_us":1374,"threads_started":1}
W20260928 16:44:35.899130 3986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:35.888149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:47135 (local address 127.26.101.62:46567)
0928 16:44:35.889101 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:35.889106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:35.889595 (+ 489us) server_negotiation.cc:407] Connection header received
0928 16:44:35.889788 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:35.889792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:35.889867 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:35.889967 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:35.890922 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.891805 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893116 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.893626 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893743 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.898829 (+ 5086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:47135: BlockingRecv error: recv got EOF from 127.26.101.1:47135 (error 108)
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":851,"spinlock_wait_cycles":6400,"thread_start_us":853,"threads_started":1}
W20260928 16:44:35.899217 3985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:35.887986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:58771 (local address 127.26.101.62:46567)
0928 16:44:35.888725 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:35.888734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:35.889187 (+ 453us) server_negotiation.cc:407] Connection header received
0928 16:44:35.889633 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:35.889638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:35.889714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:35.889799 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:35.890924 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.891866 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893110 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:35.893582 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:35.893671 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:35.898835 (+ 5164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:58771: BlockingRecv error: recv got EOF from 127.26.101.3:58771 (error 108)
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":580,"spinlock_wait_cycles":2432,"thread_start_us":361,"threads_started":1}
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:36.193686 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.184078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:35345)
0928 16:44:36.184658 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.184675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.184807 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.185169 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.185175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.185584 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.185771 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.185782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.186939 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.186944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.188415 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.188431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.189234 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.189239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.189361 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.189998 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.190016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.193416 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":338,"threads_started":1}
W20260928 16:44:36.193861 3985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:36.184342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:35345 (local address 127.26.101.62:46567)
0928 16:44:36.184591 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.184597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.184757 (+ 160us) server_negotiation.cc:407] Connection header received
0928 16:44:36.184913 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.184917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.184985 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.185073 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.185945 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.186765 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.188570 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.189006 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.189119 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.193619 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:35345: BlockingRecv error: recv got EOF from 127.26.101.2:35345 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:36 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:36.913430 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.900311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:48703)
0928 16:44:36.900990 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.901008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.901123 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.901448 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.901453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.901831 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.901993 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.902003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.903061 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.903065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.904103 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.904116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.904872 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.905043 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.909205 (+ 4162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.909231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.913095 (+ 3864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":467,"threads_started":1}
I20260928 16:44:36.913703 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:36.899860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:45303)
0928 16:44:36.900434 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:36.900456 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:36.900600 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:36.901172 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:36.901177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:36.901696 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:36.901889 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.901898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.903000 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.903006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.904002 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:36.904015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.904801 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:36.904983 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.909205 (+ 4222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:36.909231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:36.913387 (+ 4156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":318,"threads_started":1}
W20260928 16:44:36.913926 3989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:36.900113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45303 (local address 127.26.101.62:46567)
0928 16:44:36.900812 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.900822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.900841 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:44:36.900918 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.900923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.901012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.901129 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.902040 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.902869 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904148 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.904646 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904754 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.913660 (+ 8906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:45303: BlockingRecv error: recv got EOF from 127.26.101.1:45303 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":590,"thread_start_us":413,"threads_started":1}
W20260928 16:44:36.914103 3991 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:36.900646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:48703 (local address 127.26.101.62:46567)
0928 16:44:36.901183 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:36.901189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:36.901206 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:44:36.901249 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:36.901252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:36.901306 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:36.901389 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:36.902140 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.902926 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904248 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:36.904724 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:36.904835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:36.913336 (+ 8501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:48703: BlockingRecv error: recv got EOF from 127.26.101.3:48703 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:37.203758 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:37.194690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:36641)
0928 16:44:37.195298 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:37.195315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:37.195435 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:37.195705 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:37.195708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:37.196106 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:37.196326 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.196337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.197325 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.197329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.198282 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.198296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.198939 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.198944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.199127 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.199768 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:37.199786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:37.203450 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
W20260928 16:44:37.203945 3991 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:37.194773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36641 (local address 127.26.101.62:46567)
0928 16:44:37.194944 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:37.194948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:37.195401 (+ 453us) server_negotiation.cc:407] Connection header received
0928 16:44:37.195524 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:37.195528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:37.195593 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:37.195672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:37.196441 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.197163 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.198400 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.198790 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.198881 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.203705 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:36641: BlockingRecv error: recv got EOF from 127.26.101.2:36641 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:37 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:37.937263 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:37.914467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:50133)
0928 16:44:37.927708 (+ 13241us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:37.927732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:37.927890 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:37.928538 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:37.928543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:37.929021 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:37.929233 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.929248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.930575 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.930581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.931579 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.931593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.932456 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.932463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.932655 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.933291 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:37.933315 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:37.936972 (+ 3657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13093,"thread_start_us":12955,"threads_started":1}
W20260928 16:44:37.937480 3994 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:37.914558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:50133 (local address 127.26.101.62:46567)
0928 16:44:37.928047 (+ 13489us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:37.928057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:37.928079 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:44:37.928163 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:37.928183 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:37.928264 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:37.928385 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:37.929424 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.930451 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.931726 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.932305 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.932417 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.937201 (+ 4784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:50133: BlockingRecv error: recv got EOF from 127.26.101.3:50133 (error 108)
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":13357,"spinlock_wait_cycles":1792,"thread_start_us":13238,"threads_started":1}
I20260928 16:44:37.938089 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:37.914699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:57095)
0928 16:44:37.927365 (+ 12666us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:37.927393 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:37.927554 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:37.928515 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:37.928520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:37.929086 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:37.929331 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.929346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.930689 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.930693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.931685 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:37.931702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.932605 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.932613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:37.932748 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.933417 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:37.933433 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:37.937761 (+ 4328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12508,"thread_start_us":12297,"threads_started":1}
W20260928 16:44:37.938243 3996 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:37.914698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57095 (local address 127.26.101.62:46567)
0928 16:44:37.928202 (+ 13504us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:37.928207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:37.928222 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:44:37.928256 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:37.928260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:37.928307 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:37.928384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:37.929485 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.930542 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.931823 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:37.932414 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:37.932542 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:37.937989 (+ 5447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:57095: BlockingRecv error: recv got EOF from 127.26.101.1:57095 (error 108)
Metrics: {"mutex_wait_us":797,"server-negotiator.queue_time_us":13421,"spinlock_wait_cycles":15360,"thread_start_us":13309,"threads_started":1}
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:38.214380 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:38.205020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:59183)
0928 16:44:38.205673 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:38.205689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:38.205810 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:38.206173 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:38.206177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:38.206585 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:38.206758 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.206767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.207853 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.207857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.208816 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.208829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.209549 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.209556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.209737 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.210348 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:38.210366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:38.213972 (+ 3606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":365,"threads_started":1}
W20260928 16:44:38.214452 3996 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:38.205090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:59183 (local address 127.26.101.62:46567)
0928 16:44:38.205292 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:38.205298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:38.205779 (+ 481us) server_negotiation.cc:407] Connection header received
0928 16:44:38.205919 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:38.205925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:38.205994 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:38.206080 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:38.206920 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.207701 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.208960 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.209366 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.209474 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.214233 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:59183: BlockingRecv error: recv got EOF from 127.26.101.2:59183 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:38 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:38.948990 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:38.938346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:48845)
0928 16:44:38.938966 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:38.938989 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:38.939164 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:38.939675 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:38.939681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:38.940290 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:38.940507 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.940518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.941902 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.941908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.942856 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.942869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.943952 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.943959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.944168 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.944839 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:38.944863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:38.948649 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":310,"threads_started":1}
W20260928 16:44:38.949196 3999 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:38.938405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:48845 (local address 127.26.101.62:46567)
0928 16:44:38.939260 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:38.939266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:38.939282 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:44:38.939419 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:38.939423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:38.939503 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:38.939596 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:38.940674 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.941747 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.943007 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.943557 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.943679 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.948907 (+ 5228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:48845: BlockingRecv error: recv got EOF from 127.26.101.3:48845 (error 108)
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":581,"threads_started":1}
I20260928 16:44:38.949496 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:38.939220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:48463)
0928 16:44:38.940076 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:38.940118 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:38.940242 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:38.940772 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:38.940777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:38.941285 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:38.941473 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.941488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.942684 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.942688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.943844 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:38.943863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.944871 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.944876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:38.945020 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.945569 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:38.945587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:38.949187 (+ 3600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":607,"threads_started":1}
W20260928 16:44:38.949733 4001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:38.939455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:48463 (local address 127.26.101.62:46567)
0928 16:44:38.940485 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:38.940490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:38.940504 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:44:38.940547 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:38.940550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:38.940604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:38.940700 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:38.941650 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.942505 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.943999 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:38.944727 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:38.944850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:38.949472 (+ 4622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:48463: BlockingRecv error: recv got EOF from 127.26.101.1:48463 (error 108)
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":745,"threads_started":1}
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:39.224565 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.215493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:38815)
0928 16:44:39.215973 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:39.215994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:39.216132 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:39.216470 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:39.216475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:39.216928 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:39.217085 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.217096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.218130 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.218135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.219047 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.219059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.219772 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.219778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.219946 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.220565 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:39.220584 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:39.224236 (+ 3652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":229,"threads_started":1}
W20260928 16:44:39.224736 4001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:39.215710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:38815 (local address 127.26.101.62:46567)
0928 16:44:39.215944 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.215950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.216065 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:44:39.216248 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.216252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.216327 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.216413 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.217231 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.217987 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.219215 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.219629 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.219740 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.224512 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:38815: BlockingRecv error: recv got EOF from 127.26.101.2:38815 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.1@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:39 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.3@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:39.963894 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.950907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.1:47171)
0928 16:44:39.952847 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:39.952866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:39.953001 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:39.953428 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:39.953433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:39.953955 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:39.954180 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.954192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.955865 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.955870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.957391 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.957409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.958525 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.958532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.958728 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.959516 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:39.959537 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:39.963544 (+ 4007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1755,"thread_start_us":1646,"threads_started":1}
I20260928 16:44:39.963992 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:39.951159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.3:35473)
0928 16:44:39.951857 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:39.951881 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:39.952028 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:39.953441 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:39.953445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:39.953945 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:39.954153 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.954168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.955863 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.955869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.957391 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:39.957409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.958525 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.958532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:39.958718 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.959486 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:39.959510 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:39.963702 (+ 4192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":420,"threads_started":1}
W20260928 16:44:39.964066 4004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:39.951006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:47171 (local address 127.26.101.62:46567)
0928 16:44:39.953088 (+ 2082us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.953094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.953115 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:39.953172 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.953177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.953268 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.953382 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.954313 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.955691 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.957561 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.958342 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.958528 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.963795 (+ 5267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.1:47171: BlockingRecv error: recv got EOF from 127.26.101.1:47171 (error 108)
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":1434,"spinlock_wait_cycles":20096,"thread_start_us":1228,"threads_started":1}
W20260928 16:44:39.964741 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:39.952437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35473 (local address 127.26.101.62:46567)
0928 16:44:39.953112 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:39.953117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:39.953130 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:44:39.953176 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:39.953179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:39.953268 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:39.953382 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:39.954313 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.955691 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.957561 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:39.958342 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:39.958532 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:39.964434 (+ 5902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.3:35473: BlockingRecv error: recv got EOF from 127.26.101.3:35473 (error 108)
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":604,"spinlock_wait_cycles":27008,"thread_start_us":335,"threads_started":1}
Sep 28 16:44:40 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:44:40 dist-test-slave-t0tw krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.26.101.2@KRBTEST.COM for oryx/127.26.101.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:44:40.235319 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.226008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46567 (local address 127.26.101.2:40627)
0928 16:44:40.226583 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:40.226605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:40.226752 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:40.227198 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:40.227201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:40.227711 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:40.227953 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:40.227969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.229190 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.229194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:40.230145 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:40.230159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.230843 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.230847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:40.230986 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.231641 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:40.231660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:40.234969 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.26.101.62:46567: Server oryx/127.26.101.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
W20260928 16:44:40.235757 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:40.226212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:40627 (local address 127.26.101.62:46567)
0928 16:44:40.226446 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.226452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.226702 (+ 250us) server_negotiation.cc:407] Connection header received
0928 16:44:40.226890 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.226895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.227003 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.227148 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.228111 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.229020 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.230290 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.230698 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.230795 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.235366 (+ 4571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.26.101.2:40627: BlockingRecv error: recv got EOF from 127.26.101.2:40627 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 16:44:40.722590 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3513
I20260928 16:44:40.736058 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.723911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.26.101.1:36077)
0928 16:44:40.724592 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.724600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.724627 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:44:40.724697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.724706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.724967 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.725132 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.726669 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.727784 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.729042 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.729627 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.729738 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.732510 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:40.732520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:40.732538 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:40.732596 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:40.735384 (+ 2788us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:44:40.735743 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50792: Request ticket server kudu/127.26.101.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":366,"threads_started":1}
W20260928 16:44:40.736312 4009 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50792: Request ticket server kudu/127.26.101.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:44:40.736526 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3513: Not authorized: Client connection negotiation failed: client connection to 127.26.101.1:36077: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:44:40.736768 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3513
I20260928 16:44:40.751978 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3665
I20260928 16:44:40.765439 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.753226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.26.101.2:37931)
0928 16:44:40.753895 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.753902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.753931 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:44:40.754006 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.754013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.754268 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.754459 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.755870 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.756938 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.758213 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.758796 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.758918 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.761530 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:40.761538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:40.761553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:40.761600 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:40.764772 (+ 3172us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:44:40.765135 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43956: Request ticket server kudu/127.26.101.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":342,"threads_started":1}
W20260928 16:44:40.765545 4010 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43956: Request ticket server kudu/127.26.101.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:44:40.765586 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3665: Not authorized: Client connection negotiation failed: client connection to 127.26.101.2:37931: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:44:40.765770 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3665
I20260928 16:44:40.781951 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3799
I20260928 16:44:40.795055 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:40.783163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.26.101.3:34707)
0928 16:44:40.783893 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:40.783901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:40.783929 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:44:40.784000 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:40.784007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:40.784272 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:40.784472 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:40.785844 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.786869 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.788175 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:40.788744 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:40.788866 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:40.791731 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:40.791741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:40.791756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:40.791807 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:40.794308 (+ 2501us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 16:44:40.794697 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54542: Request ticket server kudu/127.26.101.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":342,"threads_started":1}
W20260928 16:44:40.795130 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3799: Not authorized: Client connection negotiation failed: client connection to 127.26.101.3:34707: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.26.101.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 16:44:40.795203 4011 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54542: Request ticket server kudu/127.26.101.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 16:44:40.795295 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3799
I20260928 16:44:40.810554 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 3443
I20260928 16:44:40.811940 3506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:40.932463 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 3443
2026-09-28T16:44:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21374 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T16:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:40Z Disabled control of system clock
I20260928 16:44:41.006507 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:34239
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:44:41.343734 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:41.344046 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:41.344127 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:41.351579 4016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:41.351703 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:41.351742 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:41.351797 4016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:41.351841 4016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:41.361420 4016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:34239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:41.362744 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:41.364622 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:41.376570 4022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.376667 4023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.377116 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.377390 4016 server_base.cc:1034] running on GCE node
I20260928 16:44:41.378264 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:41.379803 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:41.381021 4016 hybrid_clock.cc:648] HybridClock initialized: now 1790613881380961 us; error 61 us; skew 500 ppm
I20260928 16:44:41.381397 4016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:41.383880 4016 webserver.cc:460] Webserver started at http://127.26.101.62:39197/ using document root <none> and password file <none>
I20260928 16:44:41.384616 4016 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:41.384732 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:41.385064 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:41.388278 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "221f2b08258440ecacb0b1daae3e7ddc"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "914cb505798c6715563cf9ac2eb1e57b"
server_key_iv: "3d73d9e1c2d6d4c5a231fc1e39d6e354"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.389123 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "221f2b08258440ecacb0b1daae3e7ddc"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "914cb505798c6715563cf9ac2eb1e57b"
server_key_iv: "3d73d9e1c2d6d4c5a231fc1e39d6e354"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.394686 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:41.398556 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.400086 4016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:44:41.400273 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "221f2b08258440ecacb0b1daae3e7ddc"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "914cb505798c6715563cf9ac2eb1e57b"
server_key_iv: "3d73d9e1c2d6d4c5a231fc1e39d6e354"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.400497 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:41.417246 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:41.418149 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:41.418459 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:41.436353 4016 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:34239
I20260928 16:44:41.436367 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:34239 every 8 connection(s)
I20260928 16:44:41.437815 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:41.439167 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4016
I20260928 16:44:41.439368 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:41.440204 27028 external_mini_cluster.cc:1428] Setting key bb669f2f53a64d3f7c16d386049bcf51
I20260928 16:44:41.442988 4083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:41.454682 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:41.442392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.26.101.62:34239)
0928 16:44:41.443506 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:41.443518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:41.443564 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:44:41.445311 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:41.445337 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:41.445873 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:41.446271 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:41.446967 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.448460 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.449847 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:41.450549 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:41.450735 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:41.452136 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:41.452144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:41.452160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:41.452798 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:41.452880 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:41.452909 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:41.453018 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:41.453266 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 16:44:41.453608 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":366,"threads_started":1}
I20260928 16:44:41.454706 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc: Bootstrap starting.
I20260928 16:44:41.458071 4083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:41.459143 4083 log.cc:824] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:41.462641 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc: No bootstrap required, opened a new log
I20260928 16:44:41.470557 4083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } }
I20260928 16:44:41.470981 4083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:41.471101 4083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f2b08258440ecacb0b1daae3e7ddc, State: Initialized, Role: FOLLOWER
I20260928 16:44:41.471706 4083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } }
I20260928 16:44:41.471940 4083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:41.472069 4083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:41.472239 4083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:41.474517 4083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } }
I20260928 16:44:41.475047 4083 leader_election.cc:302] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221f2b08258440ecacb0b1daae3e7ddc; no voters:
I20260928 16:44:41.475746 4083 leader_election.cc:288] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:41.475917 4088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:41.477391 4088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [term 1 LEADER]: Becoming Leader. State: Replica: 221f2b08258440ecacb0b1daae3e7ddc, State: Running, Role: LEADER
I20260928 16:44:41.478022 4088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } }
I20260928 16:44:41.478472 4083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:41.481431 4089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221f2b08258440ecacb0b1daae3e7ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } } }
I20260928 16:44:41.481505 4090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221f2b08258440ecacb0b1daae3e7ddc. Latest consensus state: current_term: 1 leader_uuid: "221f2b08258440ecacb0b1daae3e7ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221f2b08258440ecacb0b1daae3e7ddc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 34239 } } }
I20260928 16:44:41.482147 4090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:41.482154 4089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:41.483633 4097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:41.488816 4097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:41.497457 4097 catalog_manager.cc:1347] Generated new cluster ID: 2e769d1f7e7e4dd78d6d51d2afdf26fa
I20260928 16:44:41.497593 4097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:41.510298 4097 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:41.511698 4097 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:41.520807 4097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221f2b08258440ecacb0b1daae3e7ddc: Generated new TSK 0
I20260928 16:44:41.521698 4097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:41.532871 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--builtin_ntp_servers=127.26.101.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 16:44:41.869427 4108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:41.869732 4108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:41.869844 4108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:41.877221 4108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:41.877333 4108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:41.877473 4108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:41.887050 4108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:41.888439 4108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:41.890354 4108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:41.904100 4114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.904269 4116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:41.904683 4113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:41.905092 4108 server_base.cc:1034] running on GCE node
I20260928 16:44:41.905815 4108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:41.906841 4108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:41.908054 4108 hybrid_clock.cc:648] HybridClock initialized: now 1790613881907975 us; error 68 us; skew 500 ppm
I20260928 16:44:41.908411 4108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:41.914628 4108 webserver.cc:460] Webserver started at http://127.26.101.1:38375/ using document root <none> and password file <none>
I20260928 16:44:41.915483 4108 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:41.915608 4108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:41.915942 4108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:41.918915 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "b7d0ee5287834ad19b3bd255107fddfa"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "8434186a1a0da16f4104698dfb2e67c1"
server_key_iv: "f3de677b89ffe3cdb62be35670901b9a"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.919809 4108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "b7d0ee5287834ad19b3bd255107fddfa"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "8434186a1a0da16f4104698dfb2e67c1"
server_key_iv: "f3de677b89ffe3cdb62be35670901b9a"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.925402 4108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:41.929171 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.930565 4108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:41.930742 4108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "b7d0ee5287834ad19b3bd255107fddfa"
format_stamp: "Formatted at 2026-09-28 16:44:41 on dist-test-slave-t0tw"
server_key: "8434186a1a0da16f4104698dfb2e67c1"
server_key_iv: "f3de677b89ffe3cdb62be35670901b9a"
server_key_version: "encryptionkey@0"
I20260928 16:44:41.930967 4108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:41.963404 4108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:41.964327 4108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:41.964619 4108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:41.965626 4108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:41.967218 4108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:41.967317 4108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.967430 4108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:41.967499 4108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:41.998713 4108 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:40911
I20260928 16:44:41.998811 4235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:40911 every 8 connection(s)
I20260928 16:44:42.000118 4108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:42.009470 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4108
I20260928 16:44:42.009693 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:42.010536 27028 external_mini_cluster.cc:1428] Setting key ae1e324030278b456b2e43a7d1044deb
I20260928 16:44:42.013573 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--builtin_ntp_servers=127.26.101.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:44:42.014454 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.002847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57959 (local address 127.26.101.62:34239)
0928 16:44:42.003391 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.003399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.004834 (+ 1435us) server_negotiation.cc:407] Connection header received
0928 16:44:42.006702 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.006707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.006788 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.006890 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.008086 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.008872 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.011408 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.011885 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.012011 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.013553 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.013557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.013561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:42.013580 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.013621 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:42.013633 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.013708 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.013980 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.014089 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
I20260928 16:44:42.015547 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.003529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:34239 (local address 127.26.101.1:57959)
0928 16:44:42.004658 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:42.004708 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:42.006416 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:42.007195 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:42.007205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:42.007305 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:42.007908 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.007926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.009066 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.009072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.011242 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.011258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.012146 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.012153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.012426 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.013190 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:42.013192 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:42.013237 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:42.013330 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:42.013724 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:42.013735 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:42.013743 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:44:42.013935 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:44:42.014370 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":297,"threads_started":1}
I20260928 16:44:42.017079 4236 heartbeater.cc:347] Connected to a master server at 127.26.101.62:34239
I20260928 16:44:42.017395 4236 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:42.018175 4236 heartbeater.cc:511] Master 127.26.101.62:34239 requested a full tablet report, sending...
I20260928 16:44:42.020401 4048 ts_manager.cc:194] Registered new tserver with Master: b7d0ee5287834ad19b3bd255107fddfa (127.26.101.1:40911)
I20260928 16:44:42.022038 4048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.1:57959
W20260928 16:44:42.353931 4240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:42.354274 4240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:42.354379 4240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:42.362197 4240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:42.362324 4240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:42.362468 4240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:42.372221 4240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:42.373601 4240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:42.375746 4240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:42.389317 4249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.389381 4245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.389343 4246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.390261 4240 server_base.cc:1034] running on GCE node
I20260928 16:44:42.390908 4240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:42.391959 4240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:42.393256 4240 hybrid_clock.cc:648] HybridClock initialized: now 1790613882393187 us; error 52 us; skew 500 ppm
I20260928 16:44:42.393636 4240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:42.396234 4240 webserver.cc:460] Webserver started at http://127.26.101.2:46795/ using document root <none> and password file <none>
I20260928 16:44:42.396963 4240 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:42.397058 4240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:42.397403 4240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:42.400468 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "feb5a092ae384976ae2cbe98e6503597"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "166cee1815f0ae6ff5487086d8035ed4"
server_key_iv: "4a42d6b91f69b37ae3795c5babc9e512"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.401291 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "feb5a092ae384976ae2cbe98e6503597"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "166cee1815f0ae6ff5487086d8035ed4"
server_key_iv: "4a42d6b91f69b37ae3795c5babc9e512"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.406970 4240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:44:42.410672 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.412298 4240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:44:42.412480 4240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "feb5a092ae384976ae2cbe98e6503597"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "166cee1815f0ae6ff5487086d8035ed4"
server_key_iv: "4a42d6b91f69b37ae3795c5babc9e512"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.412717 4240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:42.431190 4240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:42.432060 4240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:42.432351 4240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:42.433357 4240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:42.434825 4240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:42.434916 4240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.435027 4240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:42.435113 4240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.467154 4240 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:33419
I20260928 16:44:42.467252 4367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:33419 every 8 connection(s)
I20260928 16:44:42.468493 4240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:42.477494 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4240
I20260928 16:44:42.477689 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:42.478240 27028 external_mini_cluster.cc:1428] Setting key 3c46c4323fda8445df625aacf22974fe
I20260928 16:44:42.481437 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--builtin_ntp_servers=127.26.101.20:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:44:42.482026 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.471055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36631 (local address 127.26.101.62:34239)
0928 16:44:42.471284 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.471289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.472832 (+ 1543us) server_negotiation.cc:407] Connection header received
0928 16:44:42.474692 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.474698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.474782 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.474873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.476063 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.476812 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.479143 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.479697 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.479818 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.481238 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.481245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.481250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:42.481273 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.481326 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:42.481344 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.481446 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.481680 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.481771 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":98}
I20260928 16:44:42.483708 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.471666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:34239 (local address 127.26.101.2:36631)
0928 16:44:42.472626 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:42.472665 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:42.474402 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:42.475178 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:42.475187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:42.475282 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:42.475885 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.475904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.476976 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.476981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.478947 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.478966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.479923 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.479929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.480176 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.480909 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:42.480912 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:42.480952 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:42.481035 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:42.481438 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:42.481446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:42.481451 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:44:42.482213 (+ 762us) client_negotiation.cc:240] Negotiation successful
0928 16:44:42.482589 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":285,"threads_started":1}
I20260928 16:44:42.485155 4368 heartbeater.cc:347] Connected to a master server at 127.26.101.62:34239
I20260928 16:44:42.485450 4368 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:42.486231 4368 heartbeater.cc:511] Master 127.26.101.62:34239 requested a full tablet report, sending...
I20260928 16:44:42.487671 4048 ts_manager.cc:194] Registered new tserver with Master: feb5a092ae384976ae2cbe98e6503597 (127.26.101.2:33419)
I20260928 16:44:42.488481 4048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.2:36631
W20260928 16:44:42.813623 4371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:42.813903 4371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:42.813993 4371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:42.821440 4371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:42.821559 4371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:42.821672 4371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:42.830968 4371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:34239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:42.832260 4371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:42.834146 4371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:42.847157 4377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.847337 4379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:42.847183 4376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:42.848412 4371 server_base.cc:1034] running on GCE node
I20260928 16:44:42.849016 4371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:42.849956 4371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:42.851162 4371 hybrid_clock.cc:648] HybridClock initialized: now 1790613882851115 us; error 41 us; skew 500 ppm
I20260928 16:44:42.851495 4371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:42.853801 4371 webserver.cc:460] Webserver started at http://127.26.101.3:44765/ using document root <none> and password file <none>
I20260928 16:44:42.854512 4371 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:42.854602 4371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:42.854903 4371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:42.858036 4371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "303237d7995b4119a8418d33c2fe95cd"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "ab568d3c6a4617516b71cacbe9da3fec"
server_key_iv: "b0e6c32e5029f8dc29d2c95834e6749c"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.858827 4371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "303237d7995b4119a8418d33c2fe95cd"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "ab568d3c6a4617516b71cacbe9da3fec"
server_key_iv: "b0e6c32e5029f8dc29d2c95834e6749c"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.864310 4371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:44:42.867996 4385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.869499 4371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:42.869659 4371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "303237d7995b4119a8418d33c2fe95cd"
format_stamp: "Formatted at 2026-09-28 16:44:42 on dist-test-slave-t0tw"
server_key: "ab568d3c6a4617516b71cacbe9da3fec"
server_key_iv: "b0e6c32e5029f8dc29d2c95834e6749c"
server_key_version: "encryptionkey@0"
I20260928 16:44:42.869894 4371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:42.898087 4371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:42.899045 4371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:42.899435 4371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:42.900794 4371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:42.902802 4371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:42.902920 4371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:42.903038 4371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:42.903173 4371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 16:44:42.934962 4371 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:38275
I20260928 16:44:42.935051 4498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:38275 every 8 connection(s)
I20260928 16:44:42.936333 4371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:42.945461 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4371
I20260928 16:44:42.945612 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:42.946039 27028 external_mini_cluster.cc:1428] Setting key 817ca716406c3d7b415be0e1c3f015c6
I20260928 16:44:42.949937 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.938894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:35397 (local address 127.26.101.62:34239)
0928 16:44:42.939173 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.939180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.940840 (+ 1660us) server_negotiation.cc:407] Connection header received
0928 16:44:42.942755 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.942760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.942835 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.942926 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.944104 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.944892 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.946676 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.947226 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.947359 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.949165 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.949170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.949173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:42.949195 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.949231 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:42.949254 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.949365 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.949597 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.949697 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 16:44:42.951138 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.939560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:34239 (local address 127.26.101.3:35397)
0928 16:44:42.940620 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:42.940681 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:42.942462 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:42.943161 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:42.943171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:42.943305 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:42.943899 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.943917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.945057 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.945061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.946508 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:42.946527 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.947475 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.947483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:42.947817 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.948738 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:42.948742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:42.948810 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:42.948929 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:42.949380 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:42.949389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:42.949396 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:44:42.949583 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:44:42.949980 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":265,"threads_started":1}
I20260928 16:44:42.952528 4499 heartbeater.cc:347] Connected to a master server at 127.26.101.62:34239
I20260928 16:44:42.952836 4499 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:42.953629 4499 heartbeater.cc:511] Master 127.26.101.62:34239 requested a full tablet report, sending...
I20260928 16:44:42.955180 4048 ts_manager.cc:194] Registered new tserver with Master: 303237d7995b4119a8418d33c2fe95cd (127.26.101.3:38275)
I20260928 16:44:42.955961 4048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.3:35397
I20260928 16:44:42.961874 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:44:42.968122 4238 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:42.966542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.26.101.62:34239)
0928 16:44:42.966771 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.966775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.967287 (+ 512us) server_negotiation.cc:407] Connection header received
0928 16:44:42.967359 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.967364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.967456 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.967551 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.967921 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34550: BlockingRecv error: recv got EOF from 127.0.0.1:34550 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:44:42.968865 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4108
I20260928 16:44:42.977195 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:42.969895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.26.101.1:40911)
0928 16:44:42.970474 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:42.970481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:42.970508 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:44:42.970570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:42.970578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:42.970808 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:42.970964 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:42.971409 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.972445 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.973772 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:42.974538 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:42.974652 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:42.975748 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:42.975757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:42.975771 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:42.976360 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:42.976423 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:42.976443 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:42.976545 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:42.976697 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 16:44:42.976856 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260928 16:44:42.978609 4231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:43.060002 4236 heartbeater.cc:503] Master 127.26.101.62:34239 was elected leader, sending a full tablet report...
I20260928 16:44:43.061213 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4108
W20260928 16:44:43.076448 4033 connection.cc:535] server connection from 127.26.101.1:57959 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:44:43.076694 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4240
I20260928 16:44:43.085381 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:43.077658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.26.101.2:33419)
0928 16:44:43.078248 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:43.078256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:43.078304 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:44:43.078380 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:43.078387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:43.078637 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:43.078794 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:43.079180 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.080278 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.081660 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.082308 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.082418 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:43.083729 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:43.083738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:43.083753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:43.084463 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:43.084517 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:43.084541 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:43.084643 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:43.084827 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:44:43.084976 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":314,"threads_started":1}
I20260928 16:44:43.086683 4363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:43.168196 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4240
I20260928 16:44:43.183738 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4371
I20260928 16:44:43.192464 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:43.184861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.26.101.3:38275)
0928 16:44:43.185541 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:43.185548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:43.185580 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:44:43.185658 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:43.185666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:43.185923 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:43.186056 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:43.186570 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.187736 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.189018 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:43.189611 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:43.189717 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:43.190864 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:43.190873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:43.190887 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:43.191546 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:43.191593 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:43.191612 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:43.191724 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:43.191913 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:44:43.192072 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":373,"threads_started":1}
I20260928 16:44:43.193862 4494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:43.276762 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4371
I20260928 16:44:43.292910 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4016
I20260928 16:44:43.294173 4078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:43.396576 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4016
2026-09-28T16:44:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2459 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T16:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:43Z Disabled control of system clock
I20260928 16:44:43.465896 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46551
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:44:43.801698 4518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:43.802009 4518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:43.802131 4518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:43.809847 4518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:43.809986 4518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:43.810024 4518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:43.810138 4518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:43.810200 4518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:43.819696 4518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:43.821039 4518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:43.822906 4518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:43.834896 4524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:43.834901 4526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:43.835052 4523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:43.835667 4518 server_base.cc:1034] running on GCE node
I20260928 16:44:43.836483 4518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:43.837967 4518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:43.839200 4518 hybrid_clock.cc:648] HybridClock initialized: now 1790613883839139 us; error 48 us; skew 500 ppm
I20260928 16:44:43.839546 4518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:43.841993 4518 webserver.cc:460] Webserver started at http://127.26.101.62:36011/ using document root <none> and password file <none>
I20260928 16:44:43.842706 4518 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:43.842795 4518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:43.843164 4518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:43.846192 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "481480f32eeb4dee81e536648de726e9"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-t0tw"
server_key: "4ae7afd2e640c4cd58cce79b7a06f465"
server_key_iv: "1ee792ed5f8acac089ab9d3fa640316d"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.847002 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "481480f32eeb4dee81e536648de726e9"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-t0tw"
server_key: "4ae7afd2e640c4cd58cce79b7a06f465"
server_key_iv: "1ee792ed5f8acac089ab9d3fa640316d"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.852308 4518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:44:43.855942 4532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:43.857436 4518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:43.857637 4518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "481480f32eeb4dee81e536648de726e9"
format_stamp: "Formatted at 2026-09-28 16:44:43 on dist-test-slave-t0tw"
server_key: "4ae7afd2e640c4cd58cce79b7a06f465"
server_key_iv: "1ee792ed5f8acac089ab9d3fa640316d"
server_key_version: "encryptionkey@0"
I20260928 16:44:43.857880 4518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:43.863956 4518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:43.864768 4518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:43.865046 4518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:43.882562 4518 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:46551
I20260928 16:44:43.882602 4584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:46551 every 8 connection(s)
I20260928 16:44:43.883978 4518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:43.887890 4585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:43.888125 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4518
I20260928 16:44:43.888329 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:43.888947 27028 external_mini_cluster.cc:1428] Setting key 60cd85f8cc6aeee772e6cdb1502cde4f
I20260928 16:44:43.897418 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:43.890969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.26.101.62:46551)
0928 16:44:43.892047 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:43.892058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:43.892102 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:44:43.893744 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:43.893769 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:43.894228 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:43.894585 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:43.895766 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:43.895773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:43.895787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:43.895802 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:43.895873 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:43.895890 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:43.896067 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:43.896211 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 16:44:43.896493 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":365,"threads_started":1}
I20260928 16:44:43.901268 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9: Bootstrap starting.
I20260928 16:44:43.904338 4585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:43.905426 4585 log.cc:824] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:43.908456 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9: No bootstrap required, opened a new log
I20260928 16:44:43.915972 4585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } }
I20260928 16:44:43.916501 4585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:43.916612 4585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481480f32eeb4dee81e536648de726e9, State: Initialized, Role: FOLLOWER
I20260928 16:44:43.917325 4585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } }
I20260928 16:44:43.917557 4585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:43.917658 4585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:43.917790 4585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:43.920101 4585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } }
I20260928 16:44:43.920620 4585 leader_election.cc:302] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481480f32eeb4dee81e536648de726e9; no voters:
I20260928 16:44:43.921248 4585 leader_election.cc:288] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:43.921463 4590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:43.922243 4590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [term 1 LEADER]: Becoming Leader. State: Replica: 481480f32eeb4dee81e536648de726e9, State: Running, Role: LEADER
I20260928 16:44:43.922927 4590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } }
I20260928 16:44:43.923923 4585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:43.926123 4591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481480f32eeb4dee81e536648de726e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } } }
I20260928 16:44:43.926093 4592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481480f32eeb4dee81e536648de726e9. Latest consensus state: current_term: 1 leader_uuid: "481480f32eeb4dee81e536648de726e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481480f32eeb4dee81e536648de726e9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46551 } } }
I20260928 16:44:43.926654 4591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:43.926689 4592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:43.928287 4597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:43.934029 4597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:43.949863 4597 catalog_manager.cc:1347] Generated new cluster ID: ffb6b1187aed405f8ff5d6c127c34e83
I20260928 16:44:43.950007 4597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:43.967916 4597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:43.968492 4597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:43.992031 4597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 481480f32eeb4dee81e536648de726e9: Generated new TSK 0
I20260928 16:44:43.993184 4597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:44:44.008364 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--builtin_ntp_servers=127.26.101.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:44:44.353837 4609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:44.354147 4609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:44.354236 4609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:44.361941 4609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:44.362063 4609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:44.362175 4609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:44.372018 4609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:44.373335 4609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:44.375309 4609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:44.389004 4615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.388963 4617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.389034 4614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.389757 4609 server_base.cc:1034] running on GCE node
I20260928 16:44:44.390417 4609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:44.391481 4609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:44.392699 4609 hybrid_clock.cc:648] HybridClock initialized: now 1790613884392648 us; error 36 us; skew 500 ppm
I20260928 16:44:44.393047 4609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:44.395622 4609 webserver.cc:460] Webserver started at http://127.26.101.1:34959/ using document root <none> and password file <none>
I20260928 16:44:44.396466 4609 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:44.396581 4609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:44.396919 4609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:44.400125 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "984b40ffeb3b4b9ea638a4f44c2c9f89"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "9c28e8b55024b8398dd4c9272b33fb16"
server_key_iv: "492b0fe6bf01fe1e2c4cf1ad45da7831"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.400961 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "984b40ffeb3b4b9ea638a4f44c2c9f89"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "9c28e8b55024b8398dd4c9272b33fb16"
server_key_iv: "492b0fe6bf01fe1e2c4cf1ad45da7831"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.406621 4609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:44:44.410687 4623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.412240 4609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:44:44.412418 4609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "984b40ffeb3b4b9ea638a4f44c2c9f89"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "9c28e8b55024b8398dd4c9272b33fb16"
server_key_iv: "492b0fe6bf01fe1e2c4cf1ad45da7831"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.412643 4609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:44.418538 4609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:44.419420 4609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:44.419679 4609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:44.420639 4609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:44.422158 4609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:44.422258 4609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.422376 4609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:44.422452 4609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.453814 4609 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:46717
I20260928 16:44:44.453858 4736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:46717 every 8 connection(s)
I20260928 16:44:44.455257 4609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:44.462217 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4609
I20260928 16:44:44.462455 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:44.463248 27028 external_mini_cluster.cc:1428] Setting key b602c29f7a0e9213a7fee30d0119d13c
I20260928 16:44:44.464125 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.457850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:46691 (local address 127.26.101.62:46551)
0928 16:44:44.458325 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.458330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.459634 (+ 1304us) server_negotiation.cc:407] Connection header received
0928 16:44:44.461407 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.461411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.461473 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.461571 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.463304 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.463311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.463316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:44.463341 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.463403 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:44.463418 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.463533 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.463756 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.463815 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260928 16:44:44.465381 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.458453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46551 (local address 127.26.101.1:46691)
0928 16:44:44.459478 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.459521 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.461137 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.461828 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.461841 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.461981 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:44.462852 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:44.462856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:44.462915 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:44.463051 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:44.463523 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:44.463536 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:44.463543 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.463844 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.464100 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":314,"threads_started":1}
I20260928 16:44:44.466562 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--builtin_ntp_servers=127.26.101.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:44:44.467038 4737 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46551
I20260928 16:44:44.467423 4737 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:44.468155 4737 heartbeater.cc:511] Master 127.26.101.62:46551 requested a full tablet report, sending...
I20260928 16:44:44.470552 4550 ts_manager.cc:194] Registered new tserver with Master: 984b40ffeb3b4b9ea638a4f44c2c9f89 (127.26.101.1:46717)
W20260928 16:44:44.797736 4742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:44.798039 4742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:44.798132 4742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:44.805604 4742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:44.805723 4742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:44.805835 4742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:44.814987 4742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:44.816363 4742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:44.818192 4742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:44.831861 4747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.832121 4750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:44.831840 4748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:44.832410 4742 server_base.cc:1034] running on GCE node
I20260928 16:44:44.833130 4742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:44.834196 4742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:44.835427 4742 hybrid_clock.cc:648] HybridClock initialized: now 1790613884835360 us; error 60 us; skew 500 ppm
I20260928 16:44:44.835767 4742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:44.838181 4742 webserver.cc:460] Webserver started at http://127.26.101.2:35613/ using document root <none> and password file <none>
I20260928 16:44:44.838886 4742 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:44.838990 4742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:44.839357 4742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:44.842285 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "4b00f9fac13341d0bac35a64200faf80"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "1f4dacce0ff78ad93d8e4d77ab6d6a15"
server_key_iv: "0a59b461e74b8534f08d5788f283dcc4"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.843053 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "4b00f9fac13341d0bac35a64200faf80"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "1f4dacce0ff78ad93d8e4d77ab6d6a15"
server_key_iv: "0a59b461e74b8534f08d5788f283dcc4"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.848428 4742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:44.852028 4756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.853439 4742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:44.853601 4742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "4b00f9fac13341d0bac35a64200faf80"
format_stamp: "Formatted at 2026-09-28 16:44:44 on dist-test-slave-t0tw"
server_key: "1f4dacce0ff78ad93d8e4d77ab6d6a15"
server_key_iv: "0a59b461e74b8534f08d5788f283dcc4"
server_key_version: "encryptionkey@0"
I20260928 16:44:44.853816 4742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:44.859676 4742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:44.860530 4742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:44.860785 4742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:44.861807 4742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:44.863317 4742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:44.863415 4742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.863526 4742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:44.863596 4742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:44.894783 4742 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:43241
I20260928 16:44:44.894826 4869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:43241 every 8 connection(s)
I20260928 16:44:44.896288 4742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:44.900660 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4742
I20260928 16:44:44.900866 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:44.901419 27028 external_mini_cluster.cc:1428] Setting key 356786e425dda0f317a4675d8147403f
I20260928 16:44:44.904637 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--builtin_ntp_servers=127.26.101.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:44:44.905869 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.899105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:44939 (local address 127.26.101.62:46551)
0928 16:44:44.899271 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:44.899276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:44.901243 (+ 1967us) server_negotiation.cc:407] Connection header received
0928 16:44:44.903343 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:44.903349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:44.903427 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:44.903529 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:44.905128 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:44.905135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:44.905140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:44.905162 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:44.905198 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:44.905211 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:44.905317 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:44.905568 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 16:44:44.905631 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:44:44.907202 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:44.899756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46551 (local address 127.26.101.2:44939)
0928 16:44:44.901048 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:44.901099 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:44.903003 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:44.903850 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:44.903864 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:44.904005 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:44.904735 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:44.904739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:44.904795 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:44.904925 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:44.905309 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:44.905320 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:44.905329 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:44:44.905612 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 16:44:44.905921 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":388,"threads_started":1}
I20260928 16:44:44.908802 4870 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46551
I20260928 16:44:44.909132 4870 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:44.909852 4870 heartbeater.cc:511] Master 127.26.101.62:46551 requested a full tablet report, sending...
I20260928 16:44:44.911530 4550 ts_manager.cc:194] Registered new tserver with Master: 4b00f9fac13341d0bac35a64200faf80 (127.26.101.2:43241)
W20260928 16:44:45.250342 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:45.250671 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:45.250809 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:45.258483 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:45.258613 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:45.258761 4873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:45.268679 4873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:45.270045 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:45.271936 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:45.285291 4881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.285480 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:45.285239 4879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:45.286548 4873 server_base.cc:1034] running on GCE node
I20260928 16:44:45.287292 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:45.288411 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:45.289638 4873 hybrid_clock.cc:648] HybridClock initialized: now 1790613885289567 us; error 57 us; skew 500 ppm
I20260928 16:44:45.290058 4873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:44:45.293058 4873 webserver.cc:460] Webserver started at http://127.26.101.3:37357/ using document root <none> and password file <none>
I20260928 16:44:45.293768 4873 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:45.293895 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:45.294190 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:45.297209 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "21f37da4b0554202b39205d68cdcf3cd"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-t0tw"
server_key: "c15ac6ed2e634599d3738551b1974a95"
server_key_iv: "50445f81f7215d34272bfb2b7580f9ac"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.297986 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "21f37da4b0554202b39205d68cdcf3cd"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-t0tw"
server_key: "c15ac6ed2e634599d3738551b1974a95"
server_key_iv: "50445f81f7215d34272bfb2b7580f9ac"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.303678 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:44:45.307559 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.309129 4873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:45.309305 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "21f37da4b0554202b39205d68cdcf3cd"
format_stamp: "Formatted at 2026-09-28 16:44:45 on dist-test-slave-t0tw"
server_key: "c15ac6ed2e634599d3738551b1974a95"
server_key_iv: "50445f81f7215d34272bfb2b7580f9ac"
server_key_version: "encryptionkey@0"
I20260928 16:44:45.309528 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:45.315429 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:45.316356 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:45.316620 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:45.317569 4873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:45.319187 4873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:45.319279 4873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.319391 4873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:45.319458 4873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:45.350785 4873 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:35387
I20260928 16:44:45.350841 5000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:35387 every 8 connection(s)
I20260928 16:44:45.352144 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:45.359886 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 4873
I20260928 16:44:45.360091 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:45.360558 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.354732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:51017 (local address 127.26.101.62:46551)
0928 16:44:45.354928 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.354933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.356408 (+ 1475us) server_negotiation.cc:407] Connection header received
0928 16:44:45.358199 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.358205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.358321 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.358430 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.359833 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.359838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.359841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.359859 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.359897 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.359910 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.360039 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.360262 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.360324 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 16:44:45.360646 27028 external_mini_cluster.cc:1428] Setting key eb70ecc704496fb3f959af7b9bbd60bf
I20260928 16:44:45.361920 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.355287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46551 (local address 127.26.101.3:51017)
0928 16:44:45.356244 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:45.356288 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:45.357902 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:45.358692 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:45.358702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:45.358815 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:45.359449 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:44:45.359451 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:45.359499 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:44:45.359619 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:45.360012 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:45.360024 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:45.360043 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:44:45.360338 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 16:44:45.360618 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":261,"threads_started":1}
I20260928 16:44:45.363335 5001 heartbeater.cc:347] Connected to a master server at 127.26.101.62:46551
I20260928 16:44:45.363652 5001 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:45.364280 5001 heartbeater.cc:511] Master 127.26.101.62:46551 requested a full tablet report, sending...
I20260928 16:44:45.365646 4549 ts_manager.cc:194] Registered new tserver with Master: 21f37da4b0554202b39205d68cdcf3cd (127.26.101.3:35387)
I20260928 16:44:45.376588 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:44:45.382766 4740 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:44:45.381150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.26.101.62:46551)
0928 16:44:45.381349 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.381357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.381864 (+ 507us) server_negotiation.cc:407] Connection header received
0928 16:44:45.382049 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.382054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.382114 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.382205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.382496 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52878: BlockingRecv error: recv got EOF from 127.0.0.1:52878 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:44:45.383279 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4609
I20260928 16:44:45.387377 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.384488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.26.101.1:46717)
0928 16:44:45.385005 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.385016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.385051 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:44:45.385128 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.385137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.385398 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.385552 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.386587 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.386594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.386612 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.386625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.386669 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.386687 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.386822 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.386940 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.387051 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":218,"threads_started":1}
I20260928 16:44:45.388708 4732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:45.469790 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4609
I20260928 16:44:45.484323 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4742
I20260928 16:44:45.487947 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.485275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.26.101.2:43241)
0928 16:44:45.485864 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.485878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.485906 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:44:45.485973 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.485979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.486196 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.486359 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.487281 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.487289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.487302 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.487324 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.487375 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.487392 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.487498 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.487607 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.487693 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":304,"threads_started":1}
I20260928 16:44:45.489224 4865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:45.571556 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4742
I20260928 16:44:45.586812 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4873
I20260928 16:44:45.590552 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:45.587825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.26.101.3:35387)
0928 16:44:45.588457 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:45.588465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:45.588503 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:44:45.588567 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:45.588573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:45.588798 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:45.588926 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:45.589831 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:45.589838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:45.589849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:44:45.589861 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:45.589905 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:44:45.589922 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:45.590057 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:45.590114 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:44:45.590205 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":294,"threads_started":1}
I20260928 16:44:45.591907 4996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:45.674615 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4873
I20260928 16:44:45.689671 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 4518
I20260928 16:44:45.690968 4580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:45.794577 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 4518
2026-09-28T16:44:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2398 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:45 dist-test-slave-t0tw krb5kdc[5019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:45 dist-test-slave-t0tw krb5kdc[5019](info): set up 2 sockets
Sep 28 16:44:45 dist-test-slave-t0tw krb5kdc[5019](info): commencing operation
krb5kdc: starting...
W20260928 16:44:46.827664 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:46 dist-test-slave-t0tw krb5kdc[5019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:44:47.064733 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45613
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:47.393587 5033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:47.393918 5033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:47.394059 5033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:47.401440 5033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:44:47.401592 5033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:47.401679 5033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:47.401754 5033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:44:47.401811 5033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:44:47.412053 5033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:47.413434 5033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:47.414777 5033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:47.426831 5041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.426890 5039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:47.426900 5038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:47.427994 5033 server_base.cc:1034] running on GCE node
I20260928 16:44:47.428992 5033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:47.430543 5033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:47.431790 5033 hybrid_clock.cc:648] HybridClock initialized: now 1790613887431732 us; error 42 us; skew 500 ppm
I20260928 16:44:47.432183 5033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:47 dist-test-slave-t0tw krb5kdc[5019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613887, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:47.440781 5033 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:44:47.442488 5033 webserver.cc:460] Webserver started at http://127.26.101.62:42999/ using document root <none> and password file <none>
I20260928 16:44:47.443326 5033 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:47.443464 5033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:47.443776 5033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:47.447036 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "365a8bf7d10e49e998d517d38ae072bc"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-t0tw"
server_key: "9a18496228a0115ba9e4c8f25cd9b1a5"
server_key_iv: "e0950be110e7e57b924834b78723d03f"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.447875 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "365a8bf7d10e49e998d517d38ae072bc"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-t0tw"
server_key: "9a18496228a0115ba9e4c8f25cd9b1a5"
server_key_iv: "e0950be110e7e57b924834b78723d03f"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.453397 5033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:44:47.457175 5048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:47.458566 5033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:44:47.458736 5033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "365a8bf7d10e49e998d517d38ae072bc"
format_stamp: "Formatted at 2026-09-28 16:44:47 on dist-test-slave-t0tw"
server_key: "9a18496228a0115ba9e4c8f25cd9b1a5"
server_key_iv: "e0950be110e7e57b924834b78723d03f"
server_key_version: "encryptionkey@0"
I20260928 16:44:47.458930 5033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:47.475561 5033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:47.480109 5033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:47.480404 5033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:47.498179 5033 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45613
I20260928 16:44:47.498200 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45613 every 8 connection(s)
I20260928 16:44:47.499632 5033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:44:47.503489 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:47.507633 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 5033
I20260928 16:44:47.507890 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:44:47.508679 27028 external_mini_cluster.cc:1428] Setting key b0326348028a3b7183cee2d876f39b8f
I20260928 16:44:47.515590 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: Bootstrap starting.
I20260928 16:44:47.518738 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:47.519837 5100 log.cc:824] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:47.522686 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: No bootstrap required, opened a new log
Sep 28 16:44:47 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:47.530894 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } }
I20260928 16:44:47.531368 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:47.531477 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365a8bf7d10e49e998d517d38ae072bc, State: Initialized, Role: FOLLOWER
I20260928 16:44:47.531900 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:47.510965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.26.101.62:45613)
0928 16:44:47.511787 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:47.511800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:47.511843 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:44:47.514126 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:47.514180 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:47.514599 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:47.514994 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:47.516573 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.517897 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.519296 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:47.519982 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:47.520206 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:47.524225 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:47.524234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:47.524250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:47.524294 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:47.527507 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:47.528250 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:47.528258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:47.528265 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:47.528451 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:47.528842 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:47.528846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:47.528848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:47.529454 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:47.529806 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:47.530269 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 16:44:47.530713 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":345,"threads_started":1}
I20260928 16:44:47.532171 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } }
I20260928 16:44:47.532423 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:44:47.532575 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:44:47.532756 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:47.535221 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } }
I20260928 16:44:47.535756 5100 leader_election.cc:302] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365a8bf7d10e49e998d517d38ae072bc; no voters:
I20260928 16:44:47.536371 5100 leader_election.cc:288] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:44:47.536515 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:47.537139 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [term 1 LEADER]: Becoming Leader. State: Replica: 365a8bf7d10e49e998d517d38ae072bc, State: Running, Role: LEADER
I20260928 16:44:47.537672 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } }
I20260928 16:44:47.538435 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:44:47.540545 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "365a8bf7d10e49e998d517d38ae072bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } } }
I20260928 16:44:47.540910 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.541931 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365a8bf7d10e49e998d517d38ae072bc. Latest consensus state: current_term: 1 leader_uuid: "365a8bf7d10e49e998d517d38ae072bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365a8bf7d10e49e998d517d38ae072bc" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45613 } } }
I20260928 16:44:47.542294 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc [sys.catalog]: This master's current role is: LEADER
I20260928 16:44:47.544378 5111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:44:47.550163 5111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:44:47.559777 5111 catalog_manager.cc:1347] Generated new cluster ID: bb42b543084246f79c0a8de5d4fd3551
I20260928 16:44:47.559911 5111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:44:47.589181 5111 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:44:47.591225 5111 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:44:47.610952 5111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: Generated new TSK 0
I20260928 16:44:47.612210 5111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:44:47.676069 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:48.037082 5126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:48.037385 5126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:48.037510 5126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:48.045136 5126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:48.045258 5126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:48.045403 5126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:44:48.055562 5126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:48.056926 5126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:48.058280 5126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:48.072615 5134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:48.072698 5131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:48.072641 5132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:48.073407 5126 server_base.cc:1034] running on GCE node
I20260928 16:44:48.074133 5126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:48.075305 5126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:48.076546 5126 hybrid_clock.cc:648] HybridClock initialized: now 1790613888076471 us; error 70 us; skew 500 ppm
I20260928 16:44:48.076900 5126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:48 dist-test-slave-t0tw krb5kdc[5019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:48.080379 5126 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:44:48.082008 5126 webserver.cc:460] Webserver started at http://127.26.101.1:44085/ using document root <none> and password file <none>
I20260928 16:44:48.082756 5126 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:48.082883 5126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:48.083279 5126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:48.086433 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "eeb876a57dd941c0a9326daec40e079c"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "59900309342391cb5434c7ecfea2f3bf"
server_key_iv: "e71620539ce00d5d17857cf378e54182"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.087280 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb876a57dd941c0a9326daec40e079c"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "59900309342391cb5434c7ecfea2f3bf"
server_key_iv: "e71620539ce00d5d17857cf378e54182"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.092896 5126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:44:48.096872 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.098421 5126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:44:48.098599 5126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "eeb876a57dd941c0a9326daec40e079c"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "59900309342391cb5434c7ecfea2f3bf"
server_key_iv: "e71620539ce00d5d17857cf378e54182"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.098803 5126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:48.115201 5126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:48.119632 5126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:48.119928 5126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:48.120990 5126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:48.122442 5126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:48.122526 5126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.122653 5126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:48.122716 5126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.155117 5126 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:39561
I20260928 16:44:48.155165 5254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:39561 every 8 connection(s)
I20260928 16:44:48.156844 5126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:44:48.164800 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 5126
I20260928 16:44:48.165011 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:44:48.165642 27028 external_mini_cluster.cc:1428] Setting key 73ba29231e09bbe17e1eedc6d488d995
Sep 28 16:44:48 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:48.178087 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:48.159796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:43835 (local address 127.26.101.62:45613)
0928 16:44:48.160385 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:48.160391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:48.161487 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 16:44:48.163887 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:48.163893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:48.163989 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:48.164087 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:48.165802 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.166548 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.168179 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.168579 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.168703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:48.172498 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:48.172503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:48.172506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:48.172541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:48.175014 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:48.175789 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:48.175793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:48.175796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:48.175918 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:48.176343 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:48.176347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:48.176349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:48.176616 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:48.176753 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:48.177481 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 16:44:48.177720 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":262,"threads_started":1}
I20260928 16:44:48.179118 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:48.160484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45613 (local address 127.26.101.1:43835)
0928 16:44:48.161302 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:48.161350 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:48.163574 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:48.164374 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:48.164391 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:48.164937 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:48.165604 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:48.165624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.166715 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.166721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:48.168021 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:48.168034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.168979 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.168984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:48.169208 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:48.169873 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:48.169896 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:48.172261 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:48.175186 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:48.175196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:48.175215 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:48.175664 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:48.176023 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:48.176028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:48.176032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:48.176229 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:48.176746 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:48.176755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:48.177105 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 16:44:48.177351 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 16:44:48.177931 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":327,"threads_started":1}
I20260928 16:44:48.180806 5255 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45613
I20260928 16:44:48.181155 5255 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:48.182112 5255 heartbeater.cc:511] Master 127.26.101.62:45613 requested a full tablet report, sending...
I20260928 16:44:48.184697 5065 ts_manager.cc:194] Registered new tserver with Master: eeb876a57dd941c0a9326daec40e079c (127.26.101.1:39561)
I20260928 16:44:48.186197 5065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:43835
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:44:48.218534 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:48.557723 5261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:48.558027 5261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:48.558133 5261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:48.565630 5261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:48.565764 5261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:48.565881 5261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:44:48.576028 5261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:48.577373 5261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:48.578677 5261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:48.591955 5266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:48.591920 5267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:48.591882 5269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:48.592764 5261 server_base.cc:1034] running on GCE node
I20260928 16:44:48.593431 5261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:48.594436 5261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:48.595646 5261 hybrid_clock.cc:648] HybridClock initialized: now 1790613888595585 us; error 46 us; skew 500 ppm
I20260928 16:44:48.595968 5261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:48 dist-test-slave-t0tw krb5kdc[5019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:48.599207 5261 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:44:48.600663 5261 webserver.cc:460] Webserver started at http://127.26.101.2:42375/ using document root <none> and password file <none>
I20260928 16:44:48.601387 5261 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:48.601508 5261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:48.601799 5261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:48.604978 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "51b2ba3263294da1a731607f89a9ab42"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "7a953fb889b831f39a3932ad8328246d"
server_key_iv: "77a3fc19dd4e40e2711462acd5c6d17f"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.605760 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "51b2ba3263294da1a731607f89a9ab42"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "7a953fb889b831f39a3932ad8328246d"
server_key_iv: "77a3fc19dd4e40e2711462acd5c6d17f"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.611315 5261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:44:48.614926 5276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.616528 5261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:44:48.616714 5261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "51b2ba3263294da1a731607f89a9ab42"
format_stamp: "Formatted at 2026-09-28 16:44:48 on dist-test-slave-t0tw"
server_key: "7a953fb889b831f39a3932ad8328246d"
server_key_iv: "77a3fc19dd4e40e2711462acd5c6d17f"
server_key_version: "encryptionkey@0"
I20260928 16:44:48.616919 5261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:48.635234 5261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:48.639758 5261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:48.640061 5261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:48.641067 5261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:48.642506 5261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:48.642586 5261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.642704 5261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:48.642757 5261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:48.673724 5261 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:40313
I20260928 16:44:48.673786 5389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:40313 every 8 connection(s)
I20260928 16:44:48.675002 5261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:44:48.682276 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 5261
I20260928 16:44:48.682482 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:44:48.682921 27028 external_mini_cluster.cc:1428] Setting key 50bf1592a3921bd9b0131887a9020e47
Sep 28 16:44:48 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613888, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:48.696382 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:48.677600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:46539 (local address 127.26.101.62:45613)
0928 16:44:48.677780 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:48.677784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:48.678994 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 16:44:48.680970 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:48.680976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:48.681070 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:48.681161 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:48.682848 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.683635 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.685538 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.685983 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.686096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:48.690054 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:48.690060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:48.690063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:48.690101 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:48.692891 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:48.693876 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:48.693880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:48.693883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:48.694042 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:48.694562 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:48.694566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:48.694568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:48.694907 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:48.695024 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:48.695923 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 16:44:48.696120 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:48.697781 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:48.678102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45613 (local address 127.26.101.2:46539)
0928 16:44:48.678829 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:48.678876 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:48.680669 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:48.681417 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:48.681426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:48.681952 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:48.682623 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:48.682645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.683802 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.683807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:48.685390 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:48.685405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:48.686228 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:48.686236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:48.686468 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:48.687171 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:48.687203 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:48.689782 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:48.693082 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:48.693092 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:48.693112 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:48.693725 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:48.694166 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:48.694172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:48.694175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:48.694440 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:48.695031 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:48.695039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:48.695517 (+ 478us) client_negotiation.cc:769] Sending connection context
0928 16:44:48.695823 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 16:44:48.696390 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260928 16:44:48.699388 5390 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45613
I20260928 16:44:48.699694 5390 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:48.700491 5390 heartbeater.cc:511] Master 127.26.101.62:45613 requested a full tablet report, sending...
I20260928 16:44:48.702021 5065 ts_manager.cc:194] Registered new tserver with Master: 51b2ba3263294da1a731607f89a9ab42 (127.26.101.2:40313)
I20260928 16:44:48.703003 5065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:46539
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:44:48.735054 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:44:49.064713 5395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:44:49.065045 5395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:44:49.065202 5395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:44:49.072645 5395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:44:49.072796 5395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:44:49.072962 5395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:44:49.082824 5395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:44:49.084214 5395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:44:49.085623 5395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:44:49.099046 5401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.099036 5400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:44:49.098953 5403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:44:49.099932 5395 server_base.cc:1034] running on GCE node
I20260928 16:44:49.100601 5395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:44:49.101603 5395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:44:49.102836 5395 hybrid_clock.cc:648] HybridClock initialized: now 1790613889102758 us; error 69 us; skew 500 ppm
I20260928 16:44:49.103253 5395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:44:49 dist-test-slave-t0tw krb5kdc[5019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:44:49.106383 5395 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:44:49.107872 5395 webserver.cc:460] Webserver started at http://127.26.101.3:40047/ using document root <none> and password file <none>
I20260928 16:44:49.108556 5395 fs_manager.cc:359] Metadata directory not provided
I20260928 16:44:49.108662 5395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:44:49.108918 5395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:44:49.111963 5395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "187187abbf1a47108bc8751b82646eed"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-t0tw"
server_key: "d14fa6233307a001ce3cc373ece10ac4"
server_key_iv: "90389fd8b2cd47ec0c296f6cac50c70a"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.112715 5395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "187187abbf1a47108bc8751b82646eed"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-t0tw"
server_key: "d14fa6233307a001ce3cc373ece10ac4"
server_key_iv: "90389fd8b2cd47ec0c296f6cac50c70a"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.118178 5395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 16:44:49.122071 5410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.123637 5395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:44:49.123795 5395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "187187abbf1a47108bc8751b82646eed"
format_stamp: "Formatted at 2026-09-28 16:44:49 on dist-test-slave-t0tw"
server_key: "d14fa6233307a001ce3cc373ece10ac4"
server_key_iv: "90389fd8b2cd47ec0c296f6cac50c70a"
server_key_version: "encryptionkey@0"
I20260928 16:44:49.123981 5395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:44:49.160336 5395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:44:49.165086 5395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:44:49.165383 5395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:44:49.166342 5395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:44:49.167757 5395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:44:49.167837 5395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.167965 5395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:44:49.168020 5395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:44:49.189563 5255 heartbeater.cc:503] Master 127.26.101.62:45613 was elected leader, sending a full tablet report...
I20260928 16:44:49.198182 5395 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:42431
I20260928 16:44:49.198238 5523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:42431 every 8 connection(s)
I20260928 16:44:49.199584 5395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:44:49.208143 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 5395
I20260928 16:44:49.208365 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:44:49.208884 27028 external_mini_cluster.cc:1428] Setting key fb658c09192d8a2be416e959c6cb20ee
Sep 28 16:44:49 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613889, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:44:49.220495 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.202359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:47825 (local address 127.26.101.62:45613)
0928 16:44:49.202817 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.202821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.203985 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 16:44:49.205905 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.205912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.206005 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.206103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.207726 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.208873 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.210573 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.211004 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.211168 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.214616 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.214622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.214625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.214659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.217323 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.218061 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.218066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.218068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.218213 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.218646 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.218650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.218652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.218938 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.219105 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.219825 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.220093 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260928 16:44:49.221530 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.202995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45613 (local address 127.26.101.3:47825)
0928 16:44:49.203821 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.203868 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.205615 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.206361 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.206370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.206902 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:44:49.207549 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.207568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.209083 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.209089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.210414 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.210428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.211258 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.211265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.211494 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.212165 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:44:49.212193 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:44:49.214391 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:44:49.217470 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.217479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.217499 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.217932 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.218323 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:44:49.218328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:44:49.218332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:44:49.218536 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:44:49.219068 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:44:49.219108 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:44:49.219454 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.219720 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.220305 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":302,"threads_started":1}
I20260928 16:44:49.223155 5524 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45613
I20260928 16:44:49.223475 5524 heartbeater.cc:464] Registering TS with master...
I20260928 16:44:49.224442 5524 heartbeater.cc:511] Master 127.26.101.62:45613 requested a full tablet report, sending...
I20260928 16:44:49.225858 5065 ts_manager.cc:194] Registered new tserver with Master: 187187abbf1a47108bc8751b82646eed (127.26.101.3:42431)
I20260928 16:44:49.226642 5065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:47825
I20260928 16:44:49.236574 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:44:49.254129 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.241119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.26.101.62:45613)
0928 16:44:49.241348 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.241356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.241688 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:44:49.241825 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.241829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.241902 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.241992 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.243486 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.244260 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.245609 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.245999 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.246093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.248426 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.248431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.248434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.248468 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.251400 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.252116 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.252120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.252122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.252270 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.252641 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.252645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.252647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.252981 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.253222 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.253568 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.253796 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:44:49.258279 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:44:49.265321 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:44:49.304524 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.280332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.26.101.1:39561)
0928 16:44:49.295809 (+ 15477us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.295818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.296593 (+ 775us) server_negotiation.cc:407] Connection header received
0928 16:44:49.296698 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.296706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.296950 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.297141 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:49.298022 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.299320 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.301773 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.302942 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.303102 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.303367 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.303992 (+ 625us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.304170 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15307,"thread_start_us":15152,"threads_started":1}
I20260928 16:44:49.305217 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.279767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:39561 (local address 127.0.0.1:49840)
0928 16:44:49.296303 (+ 16536us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.296322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.296463 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.297392 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.297396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.297418 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:49.297829 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.297841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.299514 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.299519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.301585 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.301599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.303186 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.303193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.303421 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.303489 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.303708 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.304713 (+ 1005us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16415,"mutex_wait_us":40,"spinlock_wait_cycles":7040,"thread_start_us":16208,"threads_started":1}
I20260928 16:44:49.305341 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.278288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40313 (local address 127.0.0.1:34406)
0928 16:44:49.294627 (+ 16339us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.294713 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.294986 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.296510 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.296518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.296568 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:49.297006 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.297021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.298636 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.298643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.301285 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.301303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.303886 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.303895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.304647 (+ 752us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.304677 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.304761 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.304889 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15960,"mutex_wait_us":33,"thread_start_us":17866,"threads_started":1}
I20260928 16:44:49.306078 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.278635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.26.101.2:40313)
0928 16:44:49.295347 (+ 16712us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.295355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.295388 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:44:49.295512 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.295521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.295813 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.296036 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:49.297214 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.298477 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.301472 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.303003 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.303611 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.303822 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.305602 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.305779 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16553,"thread_start_us":16419,"threads_started":1}
I20260928 16:44:49.306620 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.278312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42431 (local address 127.0.0.1:53086)
0928 16:44:49.296301 (+ 17989us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.296328 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.296463 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.297167 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.297171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.297194 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:49.297647 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.297664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.299687 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.299693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.301865 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.301879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.304016 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.304023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.304177 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.304208 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.304335 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.306262 (+ 1927us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17836,"mutex_wait_us":35,"spinlock_wait_cycles":1280,"thread_start_us":16903,"threads_started":1}
I20260928 16:44:49.307550 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.278474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.26.101.3:42431)
0928 16:44:49.294853 (+ 16379us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.294864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.296595 (+ 1731us) server_negotiation.cc:407] Connection header received
0928 16:44:49.296698 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.296706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.296953 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.297852 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:49.298099 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.299514 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.302036 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.303525 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.303764 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.303961 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.306849 (+ 2888us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.307177 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16170,"thread_start_us":17748,"threads_started":1}
I20260928 16:44:49.310214 5325 tablet_service.cc:1467] Processing CreateTablet for tablet b5c95ca312114351bad29ee7d06aba72 (DEFAULT_TABLE table=test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:49.310737 5458 tablet_service.cc:1467] Processing CreateTablet for tablet b5c95ca312114351bad29ee7d06aba72 (DEFAULT_TABLE table=test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:49.311884 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c95ca312114351bad29ee7d06aba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:49.311874 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c95ca312114351bad29ee7d06aba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:49.312315 5190 tablet_service.cc:1467] Processing CreateTablet for tablet b5c95ca312114351bad29ee7d06aba72 (DEFAULT_TABLE table=test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:44:49.314091 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5c95ca312114351bad29ee7d06aba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:44:49.323061 5543 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Bootstrap starting.
I20260928 16:44:49.326220 5544 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Bootstrap starting.
I20260928 16:44:49.326555 5543 tablet_bootstrap.cc:654] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:49.328022 5543 log.cc:824] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:49.330163 5544 tablet_bootstrap.cc:654] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:49.331343 5543 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: No bootstrap required, opened a new log
I20260928 16:44:49.331423 5544 log.cc:824] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:49.331645 5543 ts_tablet_manager.cc:1397] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:44:49.332150 5545 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Bootstrap starting.
I20260928 16:44:49.334435 5544 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: No bootstrap required, opened a new log
I20260928 16:44:49.334702 5544 ts_tablet_manager.cc:1397] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:44:49.335623 5545 tablet_bootstrap.cc:654] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Neither blocks nor log segments found. Creating new log.
I20260928 16:44:49.336841 5545 log.cc:824] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:44:49.341933 5545 tablet_bootstrap.cc:492] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: No bootstrap required, opened a new log
I20260928 16:44:49.342247 5545 ts_tablet_manager.cc:1397] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260928 16:44:49.341974 5543 raft_consensus.cc:348] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.342424 5543 raft_consensus.cc:374] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:49.342538 5543 raft_consensus.cc:729] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187187abbf1a47108bc8751b82646eed, State: Initialized, Role: FOLLOWER
I20260928 16:44:49.342772 5544 raft_consensus.cc:348] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.343200 5544 raft_consensus.cc:374] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:49.343320 5544 raft_consensus.cc:729] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b2ba3263294da1a731607f89a9ab42, State: Initialized, Role: FOLLOWER
I20260928 16:44:49.343223 5543 consensus_queue.cc:260] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.343911 5544 consensus_queue.cc:260] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.344540 5524 heartbeater.cc:503] Master 127.26.101.62:45613 was elected leader, sending a full tablet report...
I20260928 16:44:49.345084 5543 ts_tablet_manager.cc:1428] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 16:44:49.346704 5390 heartbeater.cc:503] Master 127.26.101.62:45613 was elected leader, sending a full tablet report...
I20260928 16:44:49.347420 5544 ts_tablet_manager.cc:1428] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 16:44:49.351740 5545 raft_consensus.cc:348] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.352232 5545 raft_consensus.cc:374] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:44:49.352342 5545 raft_consensus.cc:729] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb876a57dd941c0a9326daec40e079c, State: Initialized, Role: FOLLOWER
I20260928 16:44:49.352869 5545 consensus_queue.cc:260] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.354167 5545 ts_tablet_manager.cc:1428] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 16:44:49.409669 5256 tablet.cc:2369] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:49.426992 5391 tablet.cc:2369] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:44:49.451385 5525 tablet.cc:2369] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:44:49.555851 5551 raft_consensus.cc:482] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:44:49.556180 5551 raft_consensus.cc:504] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.557943 5551 leader_election.cc:288] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 187187abbf1a47108bc8751b82646eed (127.26.101.3:42431), 51b2ba3263294da1a731607f89a9ab42 (127.26.101.2:40313)
I20260928 16:44:49.565104 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.558504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:51817 (local address 127.26.101.3:42431)
0928 16:44:49.558742 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.558747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.559529 (+ 782us) server_negotiation.cc:407] Connection header received
0928 16:44:49.559719 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.559724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.559787 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.559863 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:49.560665 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.561407 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.563290 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.564135 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.564290 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.564367 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.564734 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.564858 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 16:44:49.565142 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.558513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:60443 (local address 127.26.101.2:40313)
0928 16:44:49.558741 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.558748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.559595 (+ 847us) server_negotiation.cc:407] Connection header received
0928 16:44:49.559764 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.559769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.559833 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.559908 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:44:49.560575 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.561301 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.563303 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.564140 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.564296 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.564373 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.564758 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.564890 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 16:44:49.565304 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.558263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42431 (local address 127.26.101.1:51817)
0928 16:44:49.559400 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.559422 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.559579 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.559963 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.559969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.560013 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:49.560514 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.560525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.561554 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.561563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.563139 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.563154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.564363 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.564371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.564573 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.564603 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.564686 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.564830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":27,"thread_start_us":678,"threads_started":1}
I20260928 16:44:49.565441 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.558346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40313 (local address 127.26.101.1:60443)
0928 16:44:49.559474 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0928 16:44:49.559489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:44:49.559629 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:44:49.559934 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:44:49.559939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:44:49.559970 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:44:49.560407 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.560418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.561440 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.561444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.563113 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:44:49.563129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.564416 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.564423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:44:49.564601 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.564624 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:44:49.564704 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:44:49.564845 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":42,"thread_start_us":588,"threads_started":1}
I20260928 16:44:49.566460 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c95ca312114351bad29ee7d06aba72" candidate_uuid: "eeb876a57dd941c0a9326daec40e079c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187187abbf1a47108bc8751b82646eed" is_pre_election: true
I20260928 16:44:49.566885 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c95ca312114351bad29ee7d06aba72" candidate_uuid: "eeb876a57dd941c0a9326daec40e079c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b2ba3263294da1a731607f89a9ab42" is_pre_election: true
I20260928 16:44:49.567114 5478 raft_consensus.cc:2454] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb876a57dd941c0a9326daec40e079c in term 0.
I20260928 16:44:49.567481 5345 raft_consensus.cc:2454] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb876a57dd941c0a9326daec40e079c in term 0.
I20260928 16:44:49.567914 5145 leader_election.cc:302] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187187abbf1a47108bc8751b82646eed, eeb876a57dd941c0a9326daec40e079c; no voters:
I20260928 16:44:49.568424 5551 raft_consensus.cc:2781] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:44:49.568560 5551 raft_consensus.cc:482] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:44:49.568662 5551 raft_consensus.cc:3037] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:49.570894 5551 raft_consensus.cc:504] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.571532 5551 leader_election.cc:288] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [CANDIDATE]: Term 1 election: Requested vote from peers 187187abbf1a47108bc8751b82646eed (127.26.101.3:42431), 51b2ba3263294da1a731607f89a9ab42 (127.26.101.2:40313)
I20260928 16:44:49.572088 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c95ca312114351bad29ee7d06aba72" candidate_uuid: "eeb876a57dd941c0a9326daec40e079c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b2ba3263294da1a731607f89a9ab42"
I20260928 16:44:49.572113 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5c95ca312114351bad29ee7d06aba72" candidate_uuid: "eeb876a57dd941c0a9326daec40e079c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187187abbf1a47108bc8751b82646eed"
I20260928 16:44:49.572289 5345 raft_consensus.cc:3037] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:49.572301 5478 raft_consensus.cc:3037] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:44:49.574470 5478 raft_consensus.cc:2454] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb876a57dd941c0a9326daec40e079c in term 1.
I20260928 16:44:49.574493 5345 raft_consensus.cc:2454] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb876a57dd941c0a9326daec40e079c in term 1.
I20260928 16:44:49.574968 5143 leader_election.cc:302] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 187187abbf1a47108bc8751b82646eed, 51b2ba3263294da1a731607f89a9ab42, eeb876a57dd941c0a9326daec40e079c; no voters:
I20260928 16:44:49.575301 5551 raft_consensus.cc:2781] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:44:49.575742 5551 raft_consensus.cc:686] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 1 LEADER]: Becoming Leader. State: Replica: eeb876a57dd941c0a9326daec40e079c, State: Running, Role: LEADER
I20260928 16:44:49.576433 5551 consensus_queue.cc:237] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } }
I20260928 16:44:49.581655 5065 catalog_manager.cc:5495] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb876a57dd941c0a9326daec40e079c (127.26.101.1). New cstate: current_term: 1 leader_uuid: "eeb876a57dd941c0a9326daec40e079c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb876a57dd941c0a9326daec40e079c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39561 } health_report { overall_health: HEALTHY } } }
I20260928 16:44:49.605717 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.600257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.26.101.1:39561)
0928 16:44:49.600471 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.600477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.600498 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:49.600624 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.600628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.600722 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.600857 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:44:49.601250 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.601982 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.603532 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.603971 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.604094 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.604599 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:44:49.605125 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:44:49.605255 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.605366 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.605450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:44:49.614190 5345 raft_consensus.cc:1261] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 1 FOLLOWER]: Refusing update from remote peer eeb876a57dd941c0a9326daec40e079c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:49.614226 5478 raft_consensus.cc:1261] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 1 FOLLOWER]: Refusing update from remote peer eeb876a57dd941c0a9326daec40e079c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:44:49.615224 5551 consensus_queue.cc:1035] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [LEADER]: Connected to new peer: Peer: permanent_uuid: "187187abbf1a47108bc8751b82646eed" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:49.615553 5554 consensus_queue.cc:1035] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b2ba3263294da1a731607f89a9ab42" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:44:49.625766 5557 mvcc.cc:205] Tried to move back new op lower bound from 7334354491849965568 to 7334354491708698624. Current Snapshot: MvccSnapshot[applied={T|T < 7334354491849965568}]
I20260928 16:44:49.633044 5558 mvcc.cc:205] Tried to move back new op lower bound from 7334354491849965568 to 7334354491708698624. Current Snapshot: MvccSnapshot[applied={T|T < 7334354491849965568}]
I20260928 16:44:49.633661 5559 mvcc.cc:205] Tried to move back new op lower bound from 7334354491849965568 to 7334354491708698624. Current Snapshot: MvccSnapshot[applied={T|T < 7334354491849965568 or (T in {7334354491849965568})}]
I20260928 16:44:49.640638 5063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:49.641006 5063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:44:49.645176 5063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 365a8bf7d10e49e998d517d38ae072bc: Sending DeleteTablet for 3 replicas of tablet b5c95ca312114351bad29ee7d06aba72
I20260928 16:44:49.646507 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c95ca312114351bad29ee7d06aba72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:49 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:53086
I20260928 16:44:49.646521 5190 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c95ca312114351bad29ee7d06aba72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:49 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:49840
I20260928 16:44:49.647378 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet b5c95ca312114351bad29ee7d06aba72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:44:49 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:34406
I20260928 16:44:49.648129 5568 tablet_replica.cc:330] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: stopping tablet replica
I20260928 16:44:49.648653 5567 tablet_replica.cc:330] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: stopping tablet replica
I20260928 16:44:49.649163 5567 raft_consensus.cc:2229] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:49.649711 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 5126
I20260928 16:44:49.649730 5568 raft_consensus.cc:2229] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:44:49.650418 5567 raft_consensus.cc:2258] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:49.651278 5568 raft_consensus.cc:2258] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:49.653743 5568 ts_tablet_manager.cc:1905] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:49.658665 5567 ts_tablet_manager.cc:1905] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:49.661725 5568 ts_tablet_manager.cc:1918] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:49.662056 5568 log.cc:1196] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/b5c95ca312114351bad29ee7d06aba72
I20260928 16:44:49.662796 5568 ts_tablet_manager.cc:1939] T b5c95ca312114351bad29ee7d06aba72 P eeb876a57dd941c0a9326daec40e079c: Deleting consensus metadata
I20260928 16:44:49.665053 5051 catalog_manager.cc:4841] TS eeb876a57dd941c0a9326daec40e079c (127.26.101.1:39561): tablet b5c95ca312114351bad29ee7d06aba72 (table test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]) successfully deleted
I20260928 16:44:49.667505 5569 tablet_replica.cc:330] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: stopping tablet replica
I20260928 16:44:49.668038 5569 raft_consensus.cc:2229] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:44:49.668673 5569 raft_consensus.cc:2258] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:44:49.671793 5567 ts_tablet_manager.cc:1918] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:49.672066 5567 log.cc:1196] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/b5c95ca312114351bad29ee7d06aba72
I20260928 16:44:49.672055 5569 ts_tablet_manager.cc:1905] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:44:49.672696 5567 ts_tablet_manager.cc:1939] T b5c95ca312114351bad29ee7d06aba72 P 187187abbf1a47108bc8751b82646eed: Deleting consensus metadata
Sep 28 16:44:49 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:44:49.677312 5052 catalog_manager.cc:4841] TS 187187abbf1a47108bc8751b82646eed (127.26.101.3:42431): tablet b5c95ca312114351bad29ee7d06aba72 (table test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]) successfully deleted
I20260928 16:44:49.679543 5569 ts_tablet_manager.cc:1918] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:44:49.679704 5569 log.cc:1196] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/b5c95ca312114351bad29ee7d06aba72
I20260928 16:44:49.680279 5569 ts_tablet_manager.cc:1939] T b5c95ca312114351bad29ee7d06aba72 P 51b2ba3263294da1a731607f89a9ab42: Deleting consensus metadata
I20260928 16:44:49.682163 5050 catalog_manager.cc:4841] TS 51b2ba3263294da1a731607f89a9ab42 (127.26.101.2:40313): tablet b5c95ca312114351bad29ee7d06aba72 (table test-table [id=054f5af8e0194f38a42c7bfb6fb4db88]) successfully deleted
I20260928 16:44:49.684700 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.650929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.26.101.1:39561)
0928 16:44:49.651145 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.651151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.661443 (+ 10292us) server_negotiation.cc:407] Connection header received
0928 16:44:49.662140 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.662148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.662240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.662342 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.664402 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.665757 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.667721 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.668382 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.668490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.676185 (+ 7695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.676207 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.676225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.676282 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.680773 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.681947 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.681962 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.681965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.682174 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.682822 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.682828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.682832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.683381 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.683686 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.684064 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.684337 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:44:49.685424 5250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:49.789806 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 5126
I20260928 16:44:49.807132 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 5261
Sep 28 16:44:49 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:44:49.822702 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.807900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.26.101.2:40313)
0928 16:44:49.808073 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.808078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.808098 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:44:49.808295 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.808299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.808381 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.808464 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.809697 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.810493 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.811899 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.812305 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.812403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.816628 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.816646 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.816662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.816707 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.819576 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.820442 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.820461 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.820464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.820613 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.821049 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.821053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.821055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.821489 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.821794 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.822112 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.822364 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:44:49.823671 5385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:49.913547 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 5261
I20260928 16:44:49.928946 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 5395
Sep 28 16:44:49 dist-test-slave-t0tw krb5kdc[5019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:44:49.947189 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:44:49.929719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.26.101.3:42431)
0928 16:44:49.929897 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:44:49.929903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:44:49.929924 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:44:49.930099 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:44:49.930104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:44:49.930219 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:44:49.930311 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:44:49.931848 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.932583 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.933831 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:44:49.934225 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:44:49.934333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:44:49.938620 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:44:49.938644 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:44:49.938666 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:44:49.938711 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:44:49.943535 (+ 4824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.944651 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.944662 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.944666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.944860 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:44:49.945308 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:44:49.945313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:44:49.945317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:44:49.945838 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:44:49.946121 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 16:44:49.946587 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 16:44:49.946833 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:44:49.947937 5519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:50.053439 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 5395
I20260928 16:44:50.069223 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 5033
I20260928 16:44:50.070405 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:44:50.185985 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 5033
2026-09-28T16:44:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4392 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:44:50 dist-test-slave-t0tw krb5kdc[5577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:44:50 dist-test-slave-t0tw krb5kdc[5577](info): set up 2 sockets
Sep 28 16:44:50 dist-test-slave-t0tw krb5kdc[5577](info): commencing operation
krb5kdc: starting...
W20260928 16:44:51.223583 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:44:51 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:44:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangeradmin_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangeradmin_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangerlookup_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangerlookup_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
I20260928 16:44:51.562873 27028 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWH_khZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/postgres -l logfile start
2026-09-28 16:44:54.295 UTC [5607] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:44:54.295 UTC [5607] LOG: listening on IPv4 address "127.26.101.20", port 57529
2026-09-28 16:44:54.299 UTC [5607] LOG: listening on Unix socket "/tmp/.s.PGSQL.57529"
2026-09-28 16:44:54.320 UTC [5610] LOG: database system was shut down at 2026-09-28 16:44:52 UTC
2026-09-28 16:44:54.324 UTC [5607] LOG: database system is ready to accept connections
I20260928 16:44:55.215018 27028 mini_postgres.cc:96] Postgres bound to 57529
2026-09-28 16:44:55.234 UTC [5618] FATAL: database "slave" does not exist
127.26.101.20:57529 - accepting connections
I20260928 16:44:55.234879 27028 mini_ranger.cc:161] Starting Ranger...
I20260928 16:44:55.263412 27028 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:44:55.848055 27028 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:44:55.848234 27028 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin
2026-09-28 16:44:56,711 [I] DB FLAVOR :POSTGRES
2026-09-28 16:44:56,712 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:44:56,712 [I] Checking connection..
2026-09-28 16:44:56,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:44:57,109 [I] Checking connection passed.
2026-09-28 16:44:57,110 [I] --------- Verifying version history table ---------
2026-09-28 16:44:57,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:44:57,374 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:44:57,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:44:57,603 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:44:57,603 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:44:57,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:44:57.864 UTC [5683] WARNING: there is no transaction in progress
2026-09-28 16:44:57,870 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:44:57,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:44:58,136 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:44:58,137 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:44:58,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:44:58,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:44:58,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t0tw.c.gcp-upstream.internal','N') ;"
2026-09-28 16:44:58,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:44:59,149 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:44:59,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:44:59,419 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:44:59,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:44:59,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:44:59,941 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:44:59,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:45:01.017 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.035 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.055 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.073 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.089 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.107 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.167 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.174 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.191 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.203 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.215 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.230 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.241 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.248 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.257 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01.265 UTC [5791] WARNING: there is no transaction in progress
2026-09-28 16:45:01,996 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:45:01,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:45:02,239 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:45:02,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:45:02,498 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:45:02,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:45:02,767 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:45:02,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:45:03,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:45:03,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t0tw.c.gcp-upstream.internal';"
2026-09-28 16:45:03,509 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:45:03,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:45:03,774 [I] DB_PATCHES have already been applied
I20260928 16:45:03.780306 27028 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWH_khZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:45:03.780416 27028 mini_ranger.cc:193] Using host: 127.26.101.20
I20260928 16:45:03.795579 27028 mini_ranger.cc:227] Ranger admin URL: http://127.26.101.20:40625
Sep 28, 2026 4:45:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:45:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:45:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:45:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:45:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:45:05.832698 27028 mini_ranger.cc:160] Time spent starting Ranger: real 10.598s user 0.006s sys 0.048s
Sep 28, 2026 4:45:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:45:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:45:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:45:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:45:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:45:29.636871 27028 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangerkms_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/rangerkms_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.101.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.26.101.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:45:29 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613929, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:45:29.808962 27028 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:45:29.831707 27028 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:45:30.796816 27028 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:45:30.796988 27028 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms
I20260928 16:45:30.797127 27028 mini_ranger_kms.cc:227] Using postgres at 127.26.101.20:57529
2026-09-28 16:45:31,561 [I] DB FLAVOR :POSTGRES
2026-09-28 16:45:31,561 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:45:31,561 [I] Checking connection
2026-09-28 16:45:31,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:45:31,846 [I] connection success
2026-09-28 16:45:31,846 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:45:31,846 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:45:31,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:45:32,098 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:45:32,098 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:45:32,098 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:45:32,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:57529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:45:32,377 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:45:32.928119 27028 mini_ranger_kms.cc:310] Created kms service
I20260928 16:45:33.126091 27028 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:45:33.174338 27028 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:45:33 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613929, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:45:33.618139 27028 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:45:33.618500 27028 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:45:33.618582 27028 mini_ranger_kms.cc:240] Using host: 127.26.101.20
I20260928 16:45:33.637156 27028 mini_ranger_kms.cc:276] Ranger KMS PID: 5975
I20260928 16:45:33.637367 27028 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.26.101.20:53761
Sep 28, 2026 4:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:45:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:45:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:45:35.690446 27028 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.881s user 0.131s sys 0.433s
I20260928 16:45:35.690683 27028 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:45:35.690882 27028 mini_ranger_kms.cc:401] 127.26.101.20:53761/kms/v1/keys
Sep 28 16:45:35 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613935, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:45:36 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613935, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:45:36 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613935, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:45:39 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:45:39.299500 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41643
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:41643
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:53761/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:45:39.707021 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:39.707386 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:39.707533 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:39.715982 6026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:45:39.716106 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:39.716161 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:39.716215 6026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:45:39.716264 6026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:45:39.728186 6026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal
--ranger_kms_url=127.26.101.20:53761/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:41643
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:45:39.729807 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:39.731551 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:39.744951 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:39.745620 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:39.745854 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:39.746852 6026 server_base.cc:1034] running on GCE node
I20260928 16:45:39.747749 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:39.749436 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:39.750708 6026 hybrid_clock.cc:648] HybridClock initialized: now 1790613939750630 us; error 46 us; skew 500 ppm
I20260928 16:45:39.751147 6026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:45:39 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:39.759181 6026 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:45:39.760959 6026 webserver.cc:460] Webserver started at http://127.26.101.62:39447/ using document root <none> and password file <none>
I20260928 16:45:39.761715 6026 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:39.761844 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:39.762151 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:45:39 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:45:39.826601 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "27e636c4f4954bfb9392436ee77a1cb0"
format_stamp: "Formatted at 2026-09-28 16:45:39 on dist-test-slave-t0tw"
server_key: "3877b9875283116d51348d91d0ed002c"
server_key_iv: "7b272a0cbc3d6c518a0ef35908ac32e2"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:39.827831 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "27e636c4f4954bfb9392436ee77a1cb0"
format_stamp: "Formatted at 2026-09-28 16:45:39 on dist-test-slave-t0tw"
server_key: "3877b9875283116d51348d91d0ed002c"
server_key_iv: "7b272a0cbc3d6c518a0ef35908ac32e2"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:39.835743 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 16:45:39.858378 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:39.860351 6026 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:45:39.860584 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "27e636c4f4954bfb9392436ee77a1cb0"
format_stamp: "Formatted at 2026-09-28 16:45:39 on dist-test-slave-t0tw"
server_key: "3877b9875283116d51348d91d0ed002c"
server_key_iv: "7b272a0cbc3d6c518a0ef35908ac32e2"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:39.860867 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:39.880226 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:39.885023 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:39.885329 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:39.907883 6026 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:41643
I20260928 16:45:39.907902 6092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:41643 every 8 connection(s)
I20260928 16:45:39.909406 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:45:39.914822 6093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:39.919739 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 6026
I20260928 16:45:39.920030 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:45:39.920706 27028 external_mini_cluster.cc:1428] Setting key 125d93ad78a93b477b1ea7bbfac72a06
I20260928 16:45:39.930914 6093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: Bootstrap starting.
I20260928 16:45:39.935484 6093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:39.937006 6093 log.cc:824] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:39.942759 6093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: No bootstrap required, opened a new log
Sep 28 16:45:39 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:45:39.955139 6093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } }
I20260928 16:45:39.955734 6093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:39.955874 6093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e636c4f4954bfb9392436ee77a1cb0, State: Initialized, Role: FOLLOWER
I20260928 16:45:39.956167 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:39.922976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.26.101.62:41643)
0928 16:45:39.934008 (+ 11032us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:39.934026 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:39.934418 (+ 392us) server_negotiation.cc:407] Connection header received
0928 16:45:39.936779 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:39.936810 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:39.937297 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:39.937711 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:39.939187 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:39.941423 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:39.942752 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:39.943591 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:39.943804 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:39.947657 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:39.947666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:39.947687 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:39.947731 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:39.951237 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:39.952012 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:39.952019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:39.952027 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:39.952213 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:39.952617 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:39.952621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:39.952623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:39.953448 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:39.953832 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:39.954286 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 16:45:39.954764 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10748,"thread_start_us":10428,"threads_started":1}
I20260928 16:45:39.956693 6093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } }
I20260928 16:45:39.957067 6093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:45:39.957228 6093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:45:39.957408 6093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:39.961067 6093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } }
I20260928 16:45:39.961725 6093 leader_election.cc:302] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e636c4f4954bfb9392436ee77a1cb0; no voters:
I20260928 16:45:39.962499 6093 leader_election.cc:288] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:45:39.962613 6098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:39.963263 6098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 27e636c4f4954bfb9392436ee77a1cb0, State: Running, Role: LEADER
I20260928 16:45:39.963956 6098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } }
I20260928 16:45:39.971356 6093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:45:39.972675 6100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e636c4f4954bfb9392436ee77a1cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } } }
I20260928 16:45:39.973331 6099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e636c4f4954bfb9392436ee77a1cb0. Latest consensus state: current_term: 1 leader_uuid: "27e636c4f4954bfb9392436ee77a1cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e636c4f4954bfb9392436ee77a1cb0" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41643 } } }
I20260928 16:45:39.973673 6100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:39.973819 6099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0 [sys.catalog]: This master's current role is: LEADER
I20260928 16:45:39.980679 6103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:45:39.987149 6103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:45:39.996793 6093 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:45:40.003332 6103 catalog_manager.cc:1347] Generated new cluster ID: 933b2fe6674648fc9afae72bb09410f5
I20260928 16:45:40.004042 6103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:45:40.023998 6103 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:45:40.025952 6103 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:45:40.032179 6103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: Generated new TSK 0
I20260928 16:45:40.033095 6103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:45:40.866967 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:53761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:45:41 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:45:41.298198 6139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:41.298571 6139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:41.298736 6139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:41.307968 6139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:41.308223 6139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:41.308470 6139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:45:41.320812 6139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.26.101.20:53761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:45:41.322754 6139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:41.324208 6139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:41.352351 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:41.355558 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:41.355912 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:41.356681 6139 server_base.cc:1034] running on GCE node
I20260928 16:45:41.357515 6139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:41.358750 6139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:41.360011 6139 hybrid_clock.cc:648] HybridClock initialized: now 1790613941359940 us; error 68 us; skew 500 ppm
I20260928 16:45:41.360347 6139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:45:41 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:41.363879 6139 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:45:41.373787 6139 webserver.cc:460] Webserver started at http://127.26.101.1:33337/ using document root <none> and password file <none>
I20260928 16:45:41.374727 6139 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:41.374928 6139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:41.375409 6139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:45:41 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:45:41.395887 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "35a5f3f78d7749bd8ca916ece4b0c38e"
format_stamp: "Formatted at 2026-09-28 16:45:41 on dist-test-slave-t0tw"
server_key: "ffd809516392a318b9078be7a8bf60bc"
server_key_iv: "233652b8dacc8b3255c8d76cb13a50d4"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:41.397262 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "35a5f3f78d7749bd8ca916ece4b0c38e"
format_stamp: "Formatted at 2026-09-28 16:45:41 on dist-test-slave-t0tw"
server_key: "ffd809516392a318b9078be7a8bf60bc"
server_key_iv: "233652b8dacc8b3255c8d76cb13a50d4"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:41.406533 6139 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260928 16:45:41.430171 6159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:41.432595 6139 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260928 16:45:41.432938 6139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "35a5f3f78d7749bd8ca916ece4b0c38e"
format_stamp: "Formatted at 2026-09-28 16:45:41 on dist-test-slave-t0tw"
server_key: "ffd809516392a318b9078be7a8bf60bc"
server_key_iv: "233652b8dacc8b3255c8d76cb13a50d4"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:41.433393 6139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:41.475764 6139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:41.480801 6139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:41.481119 6139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:41.482393 6139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:41.484125 6139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:41.484212 6139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:41.484342 6139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:41.484397 6139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:41.606719 6139 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41427
I20260928 16:45:41.606729 6272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41427 every 8 connection(s)
I20260928 16:45:41.608238 6139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:45:41.609038 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 6139
I20260928 16:45:41.609277 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:45:41.610240 27028 external_mini_cluster.cc:1428] Setting key d5f2237b49b88932932da1cd82954a96
Sep 28 16:45:41 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
I20260928 16:45:41.656423 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:41.618870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:41643 (local address 127.26.101.1:59483)
0928 16:45:41.625160 (+ 6290us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:41.625225 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:41.627126 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:41.628332 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:41.628345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:41.628973 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:41.629863 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:41.629894 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:41.631506 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:41.631511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:41.633596 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:41.633628 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:41.634874 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:41.634880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:41.635200 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:41.636010 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:41.636039 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:41.643462 (+ 7423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:41.648896 (+ 5434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:41.648909 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:41.648931 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:41.650084 (+ 1153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:41.650643 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:41.650648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:41.650652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:41.650992 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:41.651921 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:41.651931 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:41.652424 (+ 493us) client_negotiation.cc:769] Sending connection context
0928 16:45:41.652924 (+ 500us) client_negotiation.cc:240] Negotiation successful
0928 16:45:41.653739 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5930,"thread_start_us":5662,"threads_started":1}
I20260928 16:45:41.657434 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:41.617964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:59483 (local address 127.26.101.62:41643)
0928 16:45:41.627741 (+ 9777us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:41.627746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:41.627763 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:45:41.627815 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:41.627819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:41.627889 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:41.627988 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:41.630132 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:41.631306 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:41.633804 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:41.634453 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:41.634566 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:41.643807 (+ 9241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:41.643815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:41.643819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:41.643884 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:41.648634 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:41.650257 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:41.650265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:41.650268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:41.650482 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:41.651337 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:41.651343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:41.651346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:41.651745 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:41.651907 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:41.656793 (+ 4886us) server_negotiation.cc:299] Negotiation successful
0928 16:45:41.657075 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9665,"thread_start_us":9451,"threads_started":1}
I20260928 16:45:41.659734 6273 heartbeater.cc:347] Connected to a master server at 127.26.101.62:41643
I20260928 16:45:41.660143 6273 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:41.661103 6273 heartbeater.cc:511] Master 127.26.101.62:41643 requested a full tablet report, sending...
I20260928 16:45:41.663978 6058 ts_manager.cc:194] Registered new tserver with Master: 35a5f3f78d7749bd8ca916ece4b0c38e (127.26.101.1:41427)
I20260928 16:45:41.666276 6058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:59483
Sep 28 16:45:41 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:45:41.687165 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:53761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:45:42 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613941, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
W20260928 16:45:42.239673 6279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:42.240235 6279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:42.240449 6279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:42.252806 6279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:42.253095 6279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:42.253407 6279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:45:42.270643 6279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.26.101.20:53761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:45:42.273036 6279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:42.274958 6279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:42.319947 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:42.322302 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:42.323745 6279 server_base.cc:1034] running on GCE node
W20260928 16:45:42.322829 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:42.325106 6279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:42.330286 6279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:42.333750 6279 hybrid_clock.cc:648] HybridClock initialized: now 1790613942333602 us; error 127 us; skew 500 ppm
I20260928 16:45:42.334357 6279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:45:42 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613942, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:42.339888 6279 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:45:42.379528 6279 webserver.cc:460] Webserver started at http://127.26.101.2:33731/ using document root <none> and password file <none>
I20260928 16:45:42.380388 6279 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:42.380537 6279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:42.380908 6279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:45:42 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613942, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:45:42.399441 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "15eb53911ec940fb8d0ef4839a3968ba"
format_stamp: "Formatted at 2026-09-28 16:45:42 on dist-test-slave-t0tw"
server_key: "7c17bf510dcabe66485e4a541097db9b"
server_key_iv: "b1924ba934708d9c6136fc6ff249b425"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:42.400722 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "15eb53911ec940fb8d0ef4839a3968ba"
format_stamp: "Formatted at 2026-09-28 16:45:42 on dist-test-slave-t0tw"
server_key: "7c17bf510dcabe66485e4a541097db9b"
server_key_iv: "b1924ba934708d9c6136fc6ff249b425"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:42.425148 6279 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.023s sys 0.000s
I20260928 16:45:42.447494 6304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:42.449667 6279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:45:42.449975 6279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "15eb53911ec940fb8d0ef4839a3968ba"
format_stamp: "Formatted at 2026-09-28 16:45:42 on dist-test-slave-t0tw"
server_key: "7c17bf510dcabe66485e4a541097db9b"
server_key_iv: "b1924ba934708d9c6136fc6ff249b425"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:42.450433 6279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:42.499289 6279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:42.505200 6279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:42.505558 6279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:42.506996 6279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:42.509238 6279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:42.509387 6279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 16:45:42.509555 6279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:42.509652 6279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:42.553145 6279 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:42399
I20260928 16:45:42.554924 6279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:45:42.553128 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:42399 every 8 connection(s)
I20260928 16:45:42.559384 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 6279
I20260928 16:45:42.559598 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:45:42.560191 27028 external_mini_cluster.cc:1428] Setting key 563d957b27e0944c6274607e3abdf1b1
Sep 28 16:45:42 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613942, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:45:42.583763 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:42.557982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:36267 (local address 127.26.101.62:41643)
0928 16:45:42.558447 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:42.558453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:42.560279 (+ 1826us) server_negotiation.cc:407] Connection header received
0928 16:45:42.562749 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:42.562755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:42.562858 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:42.562966 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:42.565543 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:42.566625 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:42.569601 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:42.570236 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:42.570366 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:42.575128 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:42.575136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:42.575141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:42.575195 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:42.579395 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:42.580527 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:42.580534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:42.580537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:42.580714 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:42.581368 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:42.581374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:42.581378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:42.581828 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:42.581998 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:42.582990 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 16:45:42.583317 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260928 16:45:42.585053 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:42.558604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:41643 (local address 127.26.101.2:36267)
0928 16:45:42.560048 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:42.560118 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:42.562411 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:42.563321 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:42.563334 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:42.564108 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:42.565252 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:42.565287 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:42.566830 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:42.566836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:42.569362 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:42.569398 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:42.570510 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:42.570518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:42.570847 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:42.571796 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:42.571827 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:42.574745 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:42.579624 (+ 4879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:42.579636 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:42.579657 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:42.580346 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:42.580845 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:42.580850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:42.580854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:42.581157 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:42.581992 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:42.582001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:42.582507 (+ 506us) client_negotiation.cc:769] Sending connection context
0928 16:45:42.582824 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 16:45:42.583463 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":577,"threads_started":1}
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:45:42.587046 6418 heartbeater.cc:347] Connected to a master server at 127.26.101.62:41643
I20260928 16:45:42.587514 6418 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:42.588443 6418 heartbeater.cc:511] Master 127.26.101.62:41643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:45:42.590327 6058 ts_manager.cc:194] Registered new tserver with Master: 15eb53911ec940fb8d0ef4839a3968ba (127.26.101.2:42399)
I20260928 16:45:42.591538 6058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:36267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:45:42.623564 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:53761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:45:42.670898 6273 heartbeater.cc:503] Master 127.26.101.62:41643 was elected leader, sending a full tablet report...
W20260928 16:45:43.013095 6424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:45:43.013405 6424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:45:43.013501 6424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:45:43.022028 6424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:45:43.022346 6424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:45:43.022620 6424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:45:43.034861 6424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.26.101.20:53761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:45:43.036207 6424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:45:43.037514 6424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:45:43.052492 6430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:43.052531 6429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:45:43.052454 6432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:45:43.053818 6424 server_base.cc:1034] running on GCE node
I20260928 16:45:43.054455 6424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:45:43.055572 6424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:45:43.056802 6424 hybrid_clock.cc:648] HybridClock initialized: now 1790613943056711 us; error 66 us; skew 500 ppm
I20260928 16:45:43.057178 6424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613943, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:45:43.060663 6424 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:45:43.062310 6424 webserver.cc:460] Webserver started at http://127.26.101.3:40515/ using document root <none> and password file <none>
I20260928 16:45:43.063315 6424 fs_manager.cc:359] Metadata directory not provided
I20260928 16:45:43.063493 6424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:45:43.063825 6424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613943, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:45:43.078297 6424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "ceb8770cf77741a8bca9b6f0d8a85d27"
format_stamp: "Formatted at 2026-09-28 16:45:43 on dist-test-slave-t0tw"
server_key: "87ab68b0b432a70eeeed3faa5b89f4d8"
server_key_iv: "35f6a570943b804b85adac45d3ee8ee0"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:43.079228 6424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "ceb8770cf77741a8bca9b6f0d8a85d27"
format_stamp: "Formatted at 2026-09-28 16:45:43 on dist-test-slave-t0tw"
server_key: "87ab68b0b432a70eeeed3faa5b89f4d8"
server_key_iv: "35f6a570943b804b85adac45d3ee8ee0"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:43.085179 6424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:45:43.101958 6439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:43.103786 6424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:45:43.103976 6424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "ceb8770cf77741a8bca9b6f0d8a85d27"
format_stamp: "Formatted at 2026-09-28 16:45:43 on dist-test-slave-t0tw"
server_key: "87ab68b0b432a70eeeed3faa5b89f4d8"
server_key_iv: "35f6a570943b804b85adac45d3ee8ee0"
server_key_version: "kuduclusterkey@0"
I20260928 16:45:43.104224 6424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:45:43.129796 6424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:45:43.134055 6424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:45:43.134357 6424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:45:43.135440 6424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:45:43.136973 6424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:45:43.137059 6424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:43.137188 6424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:45:43.137241 6424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:45:43.175688 6424 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:34907
I20260928 16:45:43.175839 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:34907 every 8 connection(s)
I20260928 16:45:43.177171 6424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:45:43.182458 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 6424
I20260928 16:45:43.182626 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:45:43.183051 27028 external_mini_cluster.cc:1428] Setting key ad81429a9e188d24c4c7158071a3def2
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613943, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:45:43.199898 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.180075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:56437 (local address 127.26.101.62:41643)
0928 16:45:43.180581 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.180585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.181549 (+ 964us) server_negotiation.cc:407] Connection header received
0928 16:45:43.183676 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.183681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.183792 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.183881 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:43.185856 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.186655 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.188382 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.188868 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.188975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.192866 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:43.192871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:43.192874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:43.192914 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:43.196155 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.197034 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.197040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.197043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.197218 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.197818 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.197822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.197824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.198227 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:43.198401 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.199230 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.199477 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":288,"threads_started":1}
I20260928 16:45:43.200873 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.180561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:41643 (local address 127.26.101.3:56437)
0928 16:45:43.181348 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.181403 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.183357 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.184194 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.184204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.184859 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:45:43.185686 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.185708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.186793 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.186797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.188205 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.188234 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.189140 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.189146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.189382 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.190093 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:45:43.190123 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:45:43.192631 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:45:43.196312 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:43.196319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:43.196336 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:43.196894 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:43.197342 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:45:43.197346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:45:43.197350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:45:43.197690 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:45:43.198357 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:45:43.198366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:45:43.198806 (+ 440us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.199058 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.199680 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":322,"threads_started":1}
I20260928 16:45:43.202373 6553 heartbeater.cc:347] Connected to a master server at 127.26.101.62:41643
I20260928 16:45:43.202750 6553 heartbeater.cc:464] Registering TS with master...
I20260928 16:45:43.203543 6553 heartbeater.cc:511] Master 127.26.101.62:41643 requested a full tablet report, sending...
I20260928 16:45:43.205171 6058 ts_manager.cc:194] Registered new tserver with Master: ceb8770cf77741a8bca9b6f0d8a85d27 (127.26.101.3:34907)
I20260928 16:45:43.206029 6058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:56437
I20260928 16:45:43.211090 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:45:43.230280 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.216053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.26.101.62:41643)
0928 16:45:43.216222 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.216226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.216693 (+ 467us) server_negotiation.cc:407] Connection header received
0928 16:45:43.216846 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.216850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.216940 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.217032 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:43.218858 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.219781 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.221254 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.221788 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.221901 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.224550 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:43.224556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:43.224559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:43.224632 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:43.227316 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.228111 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.228116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.228118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.228278 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.228772 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.228776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.228779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.229091 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:43.229294 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.229643 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.230011 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:45:43.234810 6058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:45:43.242853 6058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:45:43.279425 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.258430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:42399 (local address 127.0.0.1:52116)
0928 16:45:43.270755 (+ 12325us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.270786 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.270926 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.272153 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.272157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.272185 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:43.272610 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.272624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.274642 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.274647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.276681 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.276699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.278615 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.278632 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.278781 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.278806 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.278888 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.279022 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12175,"thread_start_us":7098,"threads_started":1}
I20260928 16:45:43.279577 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.259066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.26.101.2:42399)
0928 16:45:43.271484 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.271492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.271521 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:45:43.271620 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.271627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.271894 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.272814 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:43.273061 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.274511 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.276888 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.278320 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.278481 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.278677 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.278961 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.279222 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12233,"thread_start_us":12111,"threads_started":1}
I20260928 16:45:43.279770 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.256777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.26.101.1:41427)
0928 16:45:43.270283 (+ 13506us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.270294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.270328 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:45:43.270448 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.270457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.270774 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.271004 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:43.272317 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.273882 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.276551 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.278024 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.278179 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.278449 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.279205 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.279387 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13279,"thread_start_us":13101,"threads_started":1}
I20260928 16:45:43.280278 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.256092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41427 (local address 127.0.0.1:48732)
0928 16:45:43.266037 (+ 9945us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.266100 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.266323 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.271350 (+ 5027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.271359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.271444 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:43.272082 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.272096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.274000 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.274005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.276375 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.276389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.278252 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.278260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.278468 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.278531 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.278671 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.278902 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9609,"mutex_wait_us":1,"thread_start_us":15399,"threads_started":1}
I20260928 16:45:43.282655 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.256700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.26.101.3:34907)
0928 16:45:43.272381 (+ 15681us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.272390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.272422 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:45:43.272613 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.272621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.272880 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.273098 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:43.274111 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.275690 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.277890 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.281317 (+ 3427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.281463 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.281699 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.282144 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.282318 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15489,"thread_start_us":15289,"threads_started":1}
I20260928 16:45:43.282671 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.256362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:34907 (local address 127.0.0.1:36676)
0928 16:45:43.269387 (+ 13025us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.269413 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.269565 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.273092 (+ 3527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.273097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.273132 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:43.273788 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.273803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.275862 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.275869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.277674 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.277698 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.281641 (+ 3943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.281650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.281922 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.281959 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.282065 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.282214 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12872,"thread_start_us":12694,"threads_started":1}
I20260928 16:45:43.283983 6352 tablet_service.cc:1467] Processing CreateTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 (DEFAULT_TABLE table=test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:43.283933 6208 tablet_service.cc:1467] Processing CreateTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 (DEFAULT_TABLE table=test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:43.285328 6352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d68441b66964f1aa1e7f8e741f5a903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:43.285722 6208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d68441b66964f1aa1e7f8e741f5a903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:43.286826 6487 tablet_service.cc:1467] Processing CreateTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 (DEFAULT_TABLE table=test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:45:43.288653 6487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d68441b66964f1aa1e7f8e741f5a903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:45:43.299191 6572 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Bootstrap starting.
I20260928 16:45:43.301854 6573 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Bootstrap starting.
I20260928 16:45:43.303340 6572 tablet_bootstrap.cc:654] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:43.305014 6572 log.cc:824] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:43.306002 6573 tablet_bootstrap.cc:654] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:43.307669 6573 log.cc:824] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:43.306782 6574 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Bootstrap starting.
I20260928 16:45:43.309088 6572 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: No bootstrap required, opened a new log
I20260928 16:45:43.309370 6572 ts_tablet_manager.cc:1397] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 16:45:43.312561 6574 tablet_bootstrap.cc:654] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Neither blocks nor log segments found. Creating new log.
I20260928 16:45:43.314139 6574 log.cc:824] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Log is configured to *not* fsync() on all Append() calls
I20260928 16:45:43.316859 6573 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: No bootstrap required, opened a new log
I20260928 16:45:43.317173 6573 ts_tablet_manager.cc:1397] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260928 16:45:43.317560 6574 tablet_bootstrap.cc:492] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: No bootstrap required, opened a new log
I20260928 16:45:43.318014 6574 ts_tablet_manager.cc:1397] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 16:45:43.318890 6572 raft_consensus.cc:348] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.319418 6572 raft_consensus.cc:374] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:43.319536 6572 raft_consensus.cc:729] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15eb53911ec940fb8d0ef4839a3968ba, State: Initialized, Role: FOLLOWER
I20260928 16:45:43.320251 6572 consensus_queue.cc:260] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.321465 6418 heartbeater.cc:503] Master 127.26.101.62:41643 was elected leader, sending a full tablet report...
I20260928 16:45:43.322335 6572 ts_tablet_manager.cc:1428] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 16:45:43.326915 6573 raft_consensus.cc:348] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.327417 6573 raft_consensus.cc:374] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:43.327535 6573 raft_consensus.cc:729] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a5f3f78d7749bd8ca916ece4b0c38e, State: Initialized, Role: FOLLOWER
I20260928 16:45:43.328210 6573 consensus_queue.cc:260] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.329135 6574 raft_consensus.cc:348] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.329577 6574 raft_consensus.cc:374] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:45:43.329721 6574 raft_consensus.cc:729] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb8770cf77741a8bca9b6f0d8a85d27, State: Initialized, Role: FOLLOWER
I20260928 16:45:43.330030 6573 ts_tablet_manager.cc:1428] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 16:45:43.330374 6574 consensus_queue.cc:260] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.331526 6553 heartbeater.cc:503] Master 127.26.101.62:41643 was elected leader, sending a full tablet report...
I20260928 16:45:43.332100 6574 ts_tablet_manager.cc:1428] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
W20260928 16:45:43.370610 6274 tablet.cc:2369] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:45:43.429144 6554 tablet.cc:2369] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:45:43.442920 6579 raft_consensus.cc:482] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:45:43.443293 6579 raft_consensus.cc:504] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.445322 6579 leader_election.cc:288] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb8770cf77741a8bca9b6f0d8a85d27 (127.26.101.3:34907), 15eb53911ec940fb8d0ef4839a3968ba (127.26.101.2:42399)
I20260928 16:45:43.454350 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.446013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:39947 (local address 127.26.101.2:42399)
0928 16:45:43.446229 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.446235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.446999 (+ 764us) server_negotiation.cc:407] Connection header received
0928 16:45:43.447244 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.447251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.447342 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.447418 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:43.448064 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.449114 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.451903 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.453130 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.453307 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.453381 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.453930 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.454078 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:45:43.454350 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.445714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:34265 (local address 127.26.101.3:34907)
0928 16:45:43.446059 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.446065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.447287 (+ 1222us) server_negotiation.cc:407] Connection header received
0928 16:45:43.447478 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.447482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.447543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.447625 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:45:43.448360 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.449350 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.451903 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.453164 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.453314 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.453381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.453930 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.454100 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260928 16:45:43.454591 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.445707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:34907 (local address 127.26.101.1:34265)
0928 16:45:43.447175 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.447190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.447336 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.447661 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.447666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.447694 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:43.448183 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.448193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.449492 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.449497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.451705 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.451714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.453423 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.453430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.453710 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.453759 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.453853 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.454058 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":123,"thread_start_us":962,"threads_started":1}
I20260928 16:45:43.454596 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.445645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:42399 (local address 127.26.101.1:39947)
0928 16:45:43.446877 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0928 16:45:43.446894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:45:43.447101 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:45:43.447455 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:45:43.447460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:45:43.447485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:45:43.447915 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.447925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.449258 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.449262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.451695 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:45:43.451712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.453439 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.453444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:45:43.453704 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.453755 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 16:45:43.453853 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:45:43.454058 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":123,"thread_start_us":894,"threads_started":1}
I20260928 16:45:43.455819 6508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d68441b66964f1aa1e7f8e741f5a903" candidate_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" is_pre_election: true
I20260928 16:45:43.455875 6372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d68441b66964f1aa1e7f8e741f5a903" candidate_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15eb53911ec940fb8d0ef4839a3968ba" is_pre_election: true
I20260928 16:45:43.456347 6508 raft_consensus.cc:2454] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a5f3f78d7749bd8ca916ece4b0c38e in term 0.
I20260928 16:45:43.456346 6372 raft_consensus.cc:2454] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a5f3f78d7749bd8ca916ece4b0c38e in term 0.
I20260928 16:45:43.457059 6162 leader_election.cc:302] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a5f3f78d7749bd8ca916ece4b0c38e, ceb8770cf77741a8bca9b6f0d8a85d27; no voters:
I20260928 16:45:43.457504 6579 raft_consensus.cc:2781] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:45:43.457660 6579 raft_consensus.cc:482] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:45:43.457767 6579 raft_consensus.cc:3037] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:43.460297 6579 raft_consensus.cc:504] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.460938 6579 leader_election.cc:288] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [CANDIDATE]: Term 1 election: Requested vote from peers ceb8770cf77741a8bca9b6f0d8a85d27 (127.26.101.3:34907), 15eb53911ec940fb8d0ef4839a3968ba (127.26.101.2:42399)
I20260928 16:45:43.461546 6372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d68441b66964f1aa1e7f8e741f5a903" candidate_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15eb53911ec940fb8d0ef4839a3968ba"
I20260928 16:45:43.461680 6508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d68441b66964f1aa1e7f8e741f5a903" candidate_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27"
I20260928 16:45:43.461809 6372 raft_consensus.cc:3037] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:43.461906 6508 raft_consensus.cc:3037] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:45:43.465299 6372 raft_consensus.cc:2454] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a5f3f78d7749bd8ca916ece4b0c38e in term 1.
I20260928 16:45:43.465310 6508 raft_consensus.cc:2454] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a5f3f78d7749bd8ca916ece4b0c38e in term 1.
I20260928 16:45:43.465809 6162 leader_election.cc:302] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a5f3f78d7749bd8ca916ece4b0c38e, ceb8770cf77741a8bca9b6f0d8a85d27; no voters:
I20260928 16:45:43.466177 6579 raft_consensus.cc:2781] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:45:43.466740 6579 raft_consensus.cc:686] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 1 LEADER]: Becoming Leader. State: Replica: 35a5f3f78d7749bd8ca916ece4b0c38e, State: Running, Role: LEADER
I20260928 16:45:43.467268 6579 consensus_queue.cc:237] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } }
I20260928 16:45:43.472503 6057 catalog_manager.cc:5495] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a5f3f78d7749bd8ca916ece4b0c38e (127.26.101.1). New cstate: current_term: 1 leader_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a5f3f78d7749bd8ca916ece4b0c38e" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 } health_report { overall_health: UNKNOWN } } }
I20260928 16:45:43.508821 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.502964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.26.101.1:41427)
0928 16:45:43.503228 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.503234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.503255 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:45:43.503412 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.503416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.503497 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.503642 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:45:43.504009 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.504802 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.506319 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.506756 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.506868 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.507483 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:45:43.508180 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:45:43.508338 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.508436 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.508527 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:45:43.518188 6372 raft_consensus.cc:1261] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 1 FOLLOWER]: Refusing update from remote peer 35a5f3f78d7749bd8ca916ece4b0c38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:43.518249 6508 raft_consensus.cc:1261] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 1 FOLLOWER]: Refusing update from remote peer 35a5f3f78d7749bd8ca916ece4b0c38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:45:43.519259 6583 consensus_queue.cc:1035] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15eb53911ec940fb8d0ef4839a3968ba" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 42399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:43.519817 6579 consensus_queue.cc:1035] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb8770cf77741a8bca9b6f0d8a85d27" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:45:43.533375 6588 mvcc.cc:205] Tried to move back new op lower bound from 7334354712640126976 to 7334354712446631936. Current Snapshot: MvccSnapshot[applied={T|T < 7334354712640126976}]
I20260928 16:45:43.533934 6587 mvcc.cc:205] Tried to move back new op lower bound from 7334354712640126976 to 7334354712446631936. Current Snapshot: MvccSnapshot[applied={T|T < 7334354712640126976}]
I20260928 16:45:43.547961 6057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:43.548225 6057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:45:43.552745 6057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27e636c4f4954bfb9392436ee77a1cb0: Sending DeleteTablet for 3 replicas of tablet 0d68441b66964f1aa1e7f8e741f5a903
I20260928 16:45:43.554248 6487 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:43 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:36676
I20260928 16:45:43.554351 6352 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:43 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:52116
I20260928 16:45:43.556493 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 6139
I20260928 16:45:43.560238 6208 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d68441b66964f1aa1e7f8e741f5a903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:45:43 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:48732
I20260928 16:45:43.571657 6600 tablet_replica.cc:330] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: stopping tablet replica
I20260928 16:45:43.572628 6596 tablet_replica.cc:330] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: stopping tablet replica
I20260928 16:45:43.573724 6597 tablet_replica.cc:330] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: stopping tablet replica
I20260928 16:45:43.574150 6597 raft_consensus.cc:2229] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:43.574720 6597 raft_consensus.cc:2258] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:43.575235 6600 raft_consensus.cc:2229] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:45:43.576036 6600 raft_consensus.cc:2258] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:43.573078 6596 raft_consensus.cc:2229] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:45:43.576916 6596 raft_consensus.cc:2258] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:45:43.579727 6596 ts_tablet_manager.cc:1905] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:43.579732 6600 ts_tablet_manager.cc:1905] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:45:43.579759 6597 ts_tablet_manager.cc:1905] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:45:43.587554 6596 ts_tablet_manager.cc:1918] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:43.587558 6600 ts_tablet_manager.cc:1918] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:43.587756 6596 log.cc:1196] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/0d68441b66964f1aa1e7f8e741f5a903
I20260928 16:45:43.587757 6600 log.cc:1196] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/0d68441b66964f1aa1e7f8e741f5a903
I20260928 16:45:43.588284 6600 ts_tablet_manager.cc:1939] T 0d68441b66964f1aa1e7f8e741f5a903 P 35a5f3f78d7749bd8ca916ece4b0c38e: Deleting consensus metadata
I20260928 16:45:43.588337 6596 ts_tablet_manager.cc:1939] T 0d68441b66964f1aa1e7f8e741f5a903 P 15eb53911ec940fb8d0ef4839a3968ba: Deleting consensus metadata
I20260928 16:45:43.590258 6042 catalog_manager.cc:4841] TS 15eb53911ec940fb8d0ef4839a3968ba (127.26.101.2:42399): tablet 0d68441b66964f1aa1e7f8e741f5a903 (table test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]) successfully deleted
I20260928 16:45:43.590194 6045 catalog_manager.cc:4841] TS 35a5f3f78d7749bd8ca916ece4b0c38e (127.26.101.1:41427): tablet 0d68441b66964f1aa1e7f8e741f5a903 (table test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]) successfully deleted
I20260928 16:45:43.592056 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.561216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.26.101.1:41427)
0928 16:45:43.561427 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.561434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.569507 (+ 8073us) server_negotiation.cc:407] Connection header received
0928 16:45:43.570033 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.570041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.570137 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.570228 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:43.572159 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.573371 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.575407 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.576103 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.576240 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.582934 (+ 6694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:43.582959 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:43.582979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:43.583034 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:43.587159 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.588875 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.588885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.588889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.589078 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.589538 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.589543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.589547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.590363 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:43.590656 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.591404 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.591687 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:45:43.592978 6268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:45:43.595250 6597 ts_tablet_manager.cc:1918] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:45:43.595484 6597 log.cc:1196] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/0d68441b66964f1aa1e7f8e741f5a903
I20260928 16:45:43.596019 6597 ts_tablet_manager.cc:1939] T 0d68441b66964f1aa1e7f8e741f5a903 P ceb8770cf77741a8bca9b6f0d8a85d27: Deleting consensus metadata
I20260928 16:45:43.598074 6044 catalog_manager.cc:4841] TS ceb8770cf77741a8bca9b6f0d8a85d27 (127.26.101.3:34907): tablet 0d68441b66964f1aa1e7f8e741f5a903 (table test-table [id=63ca45217b7d4a3f8b4fd869550a90e0]) successfully deleted
I20260928 16:45:43.690591 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 6139
I20260928 16:45:43.707559 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 6279
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:45:43.722865 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.708344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.26.101.2:42399)
0928 16:45:43.708505 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.708510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.708530 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:45:43.708691 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.708697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.708784 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.708876 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:43.710334 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.711067 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.712471 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.712887 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.712987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.717267 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:43.717285 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:43.717303 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:43.717346 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:43.720227 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.720968 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.720976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.720979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.721134 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.721491 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.721495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.721497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.721890 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:43.722138 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.722401 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.722585 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:45:43.723587 6413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:45:43.815064 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 6279
I20260928 16:45:43.830236 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 6424
Sep 28 16:45:43 dist-test-slave-t0tw krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:45:43.845831 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:45:43.831136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.26.101.3:34907)
0928 16:45:43.831305 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:45:43.831309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:45:43.831329 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:45:43.831479 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:45:43.831483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:45:43.831555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:45:43.831642 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:45:43.832947 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.833696 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.835114 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:45:43.835546 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:45:43.835654 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:45:43.840253 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:45:43.840271 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:45:43.840287 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:45:43.840325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:45:43.843037 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.843830 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.843838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.843840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.843975 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:45:43.844375 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:45:43.844378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:45:43.844380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:45:43.844758 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:45:43.844972 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:45:43.845359 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 16:45:43.845528 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:45:43.846464 6548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:45:43.947453 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 6424
I20260928 16:45:43.962641 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 6026
I20260928 16:45:43.963964 6088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:45:44.077746 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 6026
I20260928 16:45:44.098392 27028 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:45:44.434834 27028 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:45:44.435145 27028 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:45:44.783010 27028 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:45:44.783 UTC [5607] LOG: received smart shutdown request
2026-09-28 16:45:44.787 UTC [5607] LOG: background worker "logical replication launcher" (PID 5616) exited with exit code 1
2026-09-28 16:45:44.787 UTC [5611] LOG: shutting down
2026-09-28 16:45:44.968 UTC [5607] LOG: database system is shut down
2026-09-28T16:45:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54904 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:45:45 dist-test-slave-t0tw krb5kdc[6613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:45:45 dist-test-slave-t0tw krb5kdc[6613](info): set up 2 sockets
Sep 28 16:45:45 dist-test-slave-t0tw krb5kdc[6613](info): commencing operation
krb5kdc: starting...
W20260928 16:45:46.175064 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:45:46 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:45:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangeradmin_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangeradmin_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangerlookup_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangerlookup_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
I20260928 16:45:46.510569 27028 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWH_khZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/postgres -l logfile start
2026-09-28 16:45:49.166 UTC [6643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:45:49.166 UTC [6643] LOG: listening on IPv4 address "127.26.101.20", port 52669
2026-09-28 16:45:49.170 UTC [6643] LOG: listening on Unix socket "/tmp/.s.PGSQL.52669"
2026-09-28 16:45:49.190 UTC [6646] LOG: database system was shut down at 2026-09-28 16:45:47 UTC
2026-09-28 16:45:49.194 UTC [6643] LOG: database system is ready to accept connections
I20260928 16:45:50.102576 27028 mini_postgres.cc:96] Postgres bound to 52669
2026-09-28 16:45:50.120 UTC [6654] FATAL: database "slave" does not exist
127.26.101.20:52669 - accepting connections
I20260928 16:45:50.120698 27028 mini_ranger.cc:161] Starting Ranger...
I20260928 16:45:50.147465 27028 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:45:50.676177 27028 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:45:50.676308 27028 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin
2026-09-28 16:45:50,961 [I] DB FLAVOR :POSTGRES
2026-09-28 16:45:50,962 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:45:50,962 [I] Checking connection..
2026-09-28 16:45:50,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:45:51,203 [I] Checking connection passed.
2026-09-28 16:45:51,204 [I] --------- Verifying version history table ---------
2026-09-28 16:45:51,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:51,468 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:45:51,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:51,713 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:45:51,713 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:45:51,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:45:51.948 UTC [6719] WARNING: there is no transaction in progress
2026-09-28 16:45:51,955 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:45:51,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:45:52,204 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:45:52,205 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:45:52,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:45:52,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:45:52,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t0tw.c.gcp-upstream.internal','N') ;"
2026-09-28 16:45:52,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:45:53,226 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:45:53,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:45:53,476 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:45:53,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:45:53,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:45:53,976 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:45:53,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:45:54.980 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:54.999 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.018 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.036 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.057 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.080 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.148 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.155 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.172 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.184 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.196 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.211 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.223 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.230 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.237 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55.245 UTC [6828] WARNING: there is no transaction in progress
2026-09-28 16:45:55,991 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:45:55,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:45:56,229 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:45:56,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:45:56,494 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:45:56,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:45:56,735 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:45:56,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:45:56,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:45:57,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t0tw.c.gcp-upstream.internal';"
2026-09-28 16:45:57,467 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:45:57,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:45:57,728 [I] DB_PATCHES have already been applied
I20260928 16:45:57.734227 27028 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWH_khZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:45:57.734329 27028 mini_ranger.cc:193] Using host: 127.26.101.20
I20260928 16:45:57.749104 27028 mini_ranger.cc:227] Ranger admin URL: http://127.26.101.20:37825
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:45:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:45:58.805100 27028 mini_ranger.cc:160] Time spent starting Ranger: real 8.684s user 0.000s sys 0.043s
Sep 28, 2026 4:46:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:46:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:46:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:46:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:46:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:46:21.430275 27028 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.26.101.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.26.101.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangerkms_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.26.101.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/rangerkms_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.26.101.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.26.101.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.26.101.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/HTTP_127.26.101.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:46:21 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613981, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:46:21.601120 27028 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:46:21.621790 27028 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:46:22.582386 27028 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:46:22.582505 27028 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms
I20260928 16:46:22.582542 27028 mini_ranger_kms.cc:227] Using postgres at 127.26.101.20:52669
2026-09-28 16:46:22,931 [I] DB FLAVOR :POSTGRES
2026-09-28 16:46:22,931 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:46:22,931 [I] Checking connection
2026-09-28 16:46:22,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:46:23,175 [I] connection success
2026-09-28 16:46:23,175 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:46:23,175 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:46:23,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:46:23,428 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:46:23,429 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:46:23,429 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:46:23,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.26.101.20:52669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:46:23,702 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:46:24.183616 27028 mini_ranger_kms.cc:310] Created kms service
I20260928 16:46:24.411052 27028 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:46:24.456260 27028 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:46:24 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613981, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:24.916648 27028 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:46:24.917018 27028 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWH_khZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWH_khZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:46:24.917096 27028 mini_ranger_kms.cc:240] Using host: 127.26.101.20
I20260928 16:46:24.940749 27028 mini_ranger_kms.cc:276] Ranger KMS PID: 7011
I20260928 16:46:24.940886 27028 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.26.101.20:56059
Sep 28, 2026 4:46:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:46:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:46:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:46:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:46:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:46:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 16:46:27 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:27.064836 27028 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.464s user 0.095s sys 0.277s
I20260928 16:46:27.065100 27028 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:46:27.065270 27028 mini_ranger_kms.cc:401] 127.26.101.20:56059/kms/v1/keys
Sep 28 16:46:27 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:46:27 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.26.101.20@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:46:30 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:46:30.292022 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42397
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:42397
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:30.646169 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:30.646492 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:30.646598 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:30.653957 7061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:30.654084 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:30.654137 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:30.654214 7061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:30.654289 7061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:30.664434 7061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
--ranger_kms_url=127.26.101.20:56059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:42397
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:30.665753 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:30.667135 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:30.679418 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:30.679469 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:30.679446 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:30.680064 7061 server_base.cc:1034] running on GCE node
I20260928 16:46:30.680905 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:30.682452 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:30.683704 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790613990683649 us; error 41 us; skew 500 ppm
I20260928 16:46:30.684088 7061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:30 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:30.687835 7061 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:30.689873 7061 webserver.cc:460] Webserver started at http://127.26.101.62:42817/ using document root <none> and password file <none>
I20260928 16:46:30.690841 7061 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:30.690985 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:30.691402 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:30 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:30.750583 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "cbec47b78a724c5889206903306e222c"
format_stamp: "Formatted at 2026-09-28 16:46:30 on dist-test-slave-t0tw"
server_key: "aa4186bfbb549c8eedaf8228c19aad61"
server_key_iv: "47fec9d8ddea41bbe157575607b05327"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:30.751474 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "cbec47b78a724c5889206903306e222c"
format_stamp: "Formatted at 2026-09-28 16:46:30 on dist-test-slave-t0tw"
server_key: "aa4186bfbb549c8eedaf8228c19aad61"
server_key_iv: "47fec9d8ddea41bbe157575607b05327"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:30.757189 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:46:30.777485 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:30.779304 7061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:46:30.779495 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "cbec47b78a724c5889206903306e222c"
format_stamp: "Formatted at 2026-09-28 16:46:30 on dist-test-slave-t0tw"
server_key: "aa4186bfbb549c8eedaf8228c19aad61"
server_key_iv: "47fec9d8ddea41bbe157575607b05327"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:30.779757 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:30.809914 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:30.814240 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:30.814710 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:30.833762 7061 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:42397
I20260928 16:46:30.833775 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:42397 every 8 connection(s)
I20260928 16:46:30.835222 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:30.839330 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:30.841394 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7061
I20260928 16:46:30.841607 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:46:30.842353 27028 external_mini_cluster.cc:1428] Setting key 806bac95917eb6a4c785a802ebb0874b
I20260928 16:46:30.851418 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Bootstrap starting.
I20260928 16:46:30.854799 7129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:30.855911 7129 log.cc:824] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:46:30 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:30.859565 7129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: No bootstrap required, opened a new log
I20260928 16:46:30.866981 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:30.844618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.26.101.62:42397)
0928 16:46:30.845469 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:30.845482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:30.845528 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:46:30.847489 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:30.847518 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:30.847964 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:30.848414 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:30.850148 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:30.851538 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:30.852932 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:30.853649 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:30.853822 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:30.857908 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:30.857921 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:30.857943 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:30.857990 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:30.861996 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:30.862841 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:30.862855 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:30.862866 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:30.863120 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:30.863541 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:30.863546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:30.863549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:30.864311 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:30.864604 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:30.864995 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 16:46:30.865538 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":379,"threads_started":1}
I20260928 16:46:30.869544 7129 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:30.869948 7129 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:30.870070 7129 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec47b78a724c5889206903306e222c, State: Initialized, Role: FOLLOWER
I20260928 16:46:30.870625 7129 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:30.870841 7129 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:30.870936 7129 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:30.871063 7129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:30.873191 7129 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:30.873690 7129 leader_election.cc:302] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbec47b78a724c5889206903306e222c; no voters:
I20260928 16:46:30.874230 7129 leader_election.cc:288] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:46:30.874356 7134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:30.875005 7134 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 LEADER]: Becoming Leader. State: Replica: cbec47b78a724c5889206903306e222c, State: Running, Role: LEADER
I20260928 16:46:30.875545 7134 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:30.876572 7129 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:30.878839 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbec47b78a724c5889206903306e222c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } } }
I20260928 16:46:30.879673 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:30.879281 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbec47b78a724c5889206903306e222c. Latest consensus state: current_term: 1 leader_uuid: "cbec47b78a724c5889206903306e222c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } } }
I20260928 16:46:30.880376 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:30.883662 7142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:30.889600 7142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:30.890277 7129 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:46:30.915654 7142 catalog_manager.cc:1347] Generated new cluster ID: fcd19216cc0940f1a3545dc28e1b6e09
I20260928 16:46:30.916584 7142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:30.943761 7142 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:30.945962 7142 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:30.959861 7142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Generated new TSK 0
I20260928 16:46:30.961107 7142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:46:31.708051 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:31 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613991, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:46:32.184976 7174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:32.185436 7174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:32.185611 7174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:32.198365 7174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:32.198637 7174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:32.198880 7174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:46:32.215561 7174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:32.217470 7174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:32.219516 7174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:32.239259 7184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:32.239176 7185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:32.241834 7188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:32.242429 7174 server_base.cc:1034] running on GCE node
I20260928 16:46:32.243224 7174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:32.244447 7174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:32.245724 7174 hybrid_clock.cc:648] HybridClock initialized: now 1790613992245654 us; error 66 us; skew 500 ppm
I20260928 16:46:32.246143 7174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:32 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613992, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:32.251179 7174 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:46:32.253739 7174 webserver.cc:460] Webserver started at http://127.26.101.1:43063/ using document root <none> and password file <none>
I20260928 16:46:32.255008 7174 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:32.255306 7174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:32.255868 7174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:32 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613992, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:32.282732 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "ef16e851d9074f26926ea41f39e86b75"
format_stamp: "Formatted at 2026-09-28 16:46:32 on dist-test-slave-t0tw"
server_key: "11affa906dda5b5942482da39b39e8ce"
server_key_iv: "2cf0e3aa39086e6b5bddc6516f0d8e01"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:32.286859 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "ef16e851d9074f26926ea41f39e86b75"
format_stamp: "Formatted at 2026-09-28 16:46:32 on dist-test-slave-t0tw"
server_key: "11affa906dda5b5942482da39b39e8ce"
server_key_iv: "2cf0e3aa39086e6b5bddc6516f0d8e01"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:32.294890 7174 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 16:46:32.314723 7195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:32.316879 7174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:46:32.317123 7174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "ef16e851d9074f26926ea41f39e86b75"
format_stamp: "Formatted at 2026-09-28 16:46:32 on dist-test-slave-t0tw"
server_key: "11affa906dda5b5942482da39b39e8ce"
server_key_iv: "2cf0e3aa39086e6b5bddc6516f0d8e01"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:32.317446 7174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:32.348726 7174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:32.355016 7174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:32.355537 7174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:32.363248 7174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:32.371944 7174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:32.372169 7174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:46:32.372419 7174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:32.372550 7174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:32.476081 7174 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:34011
I20260928 16:46:32.476135 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:34011 every 8 connection(s)
I20260928 16:46:32.478048 7174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:32.488372 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7174
I20260928 16:46:32.488602 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:46:32.489450 27028 external_mini_cluster.cc:1428] Setting key 3b85d0ba47f0717368620789b113c2e4
Sep 28 16:46:32 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613992, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:32.509732 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:32.483309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:39845 (local address 127.26.101.62:42397)
0928 16:46:32.484716 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:32.484723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:32.488509 (+ 3786us) server_negotiation.cc:407] Connection header received
0928 16:46:32.488583 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:32.488588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:32.488694 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:32.488832 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:32.491181 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:32.492402 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:32.494966 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:32.495656 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:32.495766 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:32.501350 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:32.501359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:32.501364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:32.501423 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:32.505461 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:32.506566 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:32.506572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:32.506575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:32.506747 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:32.507407 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:32.507412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:32.507415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:32.507803 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:32.507950 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:32.509048 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0928 16:46:32.509315 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":1119,"threads_started":1}
I20260928 16:46:32.512847 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:32.482245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.1:39845)
0928 16:46:32.485401 (+ 3156us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:32.485466 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:32.488071 (+ 2605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:32.489301 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:32.489313 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:32.490001 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:32.490917 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:32.490955 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:32.492631 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:32.492637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:32.494761 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:32.494779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:32.496190 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:32.496199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:32.496531 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:32.498313 (+ 1782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:32.498345 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:32.500991 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:32.505697 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:32.505708 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:32.505727 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:32.506400 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:32.506918 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:32.506922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:32.506926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:32.507257 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:32.508063 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:32.508072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:32.508563 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 16:46:32.510507 (+ 1944us) client_negotiation.cc:240] Negotiation successful
0928 16:46:32.511224 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2841,"thread_start_us":2572,"threads_started":1}
I20260928 16:46:32.515532 7311 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:32.515924 7311 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:32.516954 7311 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
Sep 28 16:46:32 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613991, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:32.520136 7094 ts_manager.cc:194] Registered new tserver with Master: ef16e851d9074f26926ea41f39e86b75 (127.26.101.1:34011)
I20260928 16:46:32.522166 7094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:39845
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:46:32.582569 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:32 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613991, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
W20260928 16:46:33.096561 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:33.097105 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:33.097335 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:33.107971 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:33.108237 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:33.108484 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:46:33.125522 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:33.127795 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:33.129866 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:33.172503 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:33.176883 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:33.180385 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:33.187826 7317 server_base.cc:1034] running on GCE node
I20260928 16:46:33.189059 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:33.203487 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:33.206970 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790613993206816 us; error 139 us; skew 500 ppm
I20260928 16:46:33.207501 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:33 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:33.212525 7317 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:46:33.247618 7317 webserver.cc:460] Webserver started at http://127.26.101.2:45777/ using document root <none> and password file <none>
I20260928 16:46:33.248863 7317 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:33.249094 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:33.249567 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:33 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:33.288954 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "99188683cb664e50b6063ade776f31f9"
format_stamp: "Formatted at 2026-09-28 16:46:33 on dist-test-slave-t0tw"
server_key: "3f8e0d07fdeb6e5fe2b4d74b894c2429"
server_key_iv: "0c65c57808c1dd321809a92110b1c7ab"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:33.290624 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "99188683cb664e50b6063ade776f31f9"
format_stamp: "Formatted at 2026-09-28 16:46:33 on dist-test-slave-t0tw"
server_key: "3f8e0d07fdeb6e5fe2b4d74b894c2429"
server_key_iv: "0c65c57808c1dd321809a92110b1c7ab"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:33.304853 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.003s
I20260928 16:46:33.332599 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:33.334501 7317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 16:46:33.334728 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "99188683cb664e50b6063ade776f31f9"
format_stamp: "Formatted at 2026-09-28 16:46:33 on dist-test-slave-t0tw"
server_key: "3f8e0d07fdeb6e5fe2b4d74b894c2429"
server_key_iv: "0c65c57808c1dd321809a92110b1c7ab"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:33.334985 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:33.353387 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:33.358078 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:33.358393 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:33.359622 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:33.361770 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:33.361935 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:33.362076 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:33.362134 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:33.421768 7317 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:40495
I20260928 16:46:33.421808 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:40495 every 8 connection(s)
I20260928 16:46:33.423422 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:33.433565 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7317
I20260928 16:46:33.433771 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:46:33.434396 27028 external_mini_cluster.cc:1428] Setting key 15a4272dd7c14475c89efd61a3660e03
Sep 28 16:46:33 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:46:33.496388 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:33.428740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58891 (local address 127.26.101.62:42397)
0928 16:46:33.464789 (+ 36049us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:33.464797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:33.469580 (+ 4783us) server_negotiation.cc:407] Connection header received
0928 16:46:33.469699 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:33.469706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:33.469812 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:33.469938 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:33.478910 (+ 8972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:33.480441 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:33.482805 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:33.483587 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:33.483694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:33.488627 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:33.488634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:33.488639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:33.488684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:33.492395 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:33.493488 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:33.493494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:33.493497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:33.493683 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:33.494311 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:33.494315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:33.494319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:33.494729 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:33.494869 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:33.495835 (+ 966us) server_negotiation.cc:299] Negotiation successful
0928 16:46:33.496047 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35868,"thread_start_us":36080,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:46:33.498802 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:33.428034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.2:58891)
0928 16:46:33.465676 (+ 37642us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:33.465754 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:33.468919 (+ 3165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:33.476889 (+ 7970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:33.476903 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:33.477730 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:33.478671 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:33.478701 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:33.480673 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:33.480678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:33.482608 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:33.482625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:33.484132 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:33.484139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:33.484502 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:33.485366 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:33.485396 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:33.488297 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:33.492609 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:33.492620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:33.492643 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:33.493307 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:33.493858 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:33.493863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:33.493866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:33.494172 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:33.494980 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:33.494986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:33.495402 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 16:46:33.496728 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0928 16:46:33.497281 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37221,"thread_start_us":46929,"threads_started":1}
I20260928 16:46:33.505144 7455 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:33.505511 7455 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:33.506390 7455 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:33.508069 7094 ts_manager.cc:194] Registered new tserver with Master: 99188683cb664e50b6063ade776f31f9 (127.26.101.2:40495)
I20260928 16:46:33.508891 7094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:58891
I20260928 16:46:33.526466 7311 heartbeater.cc:503] Master 127.26.101.62:42397 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:46:33.532323 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:33.948954 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:33.949295 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:33.949437 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:33.957489 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:33.957619 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:33.957769 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:46:33.969002 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:33.970357 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:33.971719 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:33.986353 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:33.986387 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:33.986424 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:33.987679 7462 server_base.cc:1034] running on GCE node
I20260928 16:46:33.988312 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:33.989336 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:33.990602 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790613993990538 us; error 48 us; skew 500 ppm
I20260928 16:46:33.990947 7462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:33 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:33.994207 7462 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:46:33.995973 7462 webserver.cc:460] Webserver started at http://127.26.101.3:37439/ using document root <none> and password file <none>
I20260928 16:46:33.996851 7462 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:33.997114 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:33.997500 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:46:33 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:34.014858 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "3b8439c3e6b04d95b10da263ac8a60fc"
format_stamp: "Formatted at 2026-09-28 16:46:34 on dist-test-slave-t0tw"
server_key: "3d2305bfd9fb3e423f0da6a1a838d495"
server_key_iv: "31bba40a9a2ae016b93f45dde8b0f0b3"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:34.016294 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "3b8439c3e6b04d95b10da263ac8a60fc"
format_stamp: "Formatted at 2026-09-28 16:46:34 on dist-test-slave-t0tw"
server_key: "3d2305bfd9fb3e423f0da6a1a838d495"
server_key_iv: "31bba40a9a2ae016b93f45dde8b0f0b3"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:34.024381 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260928 16:46:34.044031 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:34.046162 7462 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 16:46:34.046391 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "3b8439c3e6b04d95b10da263ac8a60fc"
format_stamp: "Formatted at 2026-09-28 16:46:34 on dist-test-slave-t0tw"
server_key: "3d2305bfd9fb3e423f0da6a1a838d495"
server_key_iv: "31bba40a9a2ae016b93f45dde8b0f0b3"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:34.046675 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:34.064885 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:34.069509 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:34.069851 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:34.071000 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:34.072502 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:34.072582 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:34.072706 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:34.072762 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:34.116014 7462 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:40819
I20260928 16:46:34.116065 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:40819 every 8 connection(s)
I20260928 16:46:34.117473 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:34.119374 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7462
I20260928 16:46:34.119540 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:46:34.120057 27028 external_mini_cluster.cc:1428] Setting key 17092f95f3d1146815278c8b8212febf
Sep 28 16:46:34 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613993, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:34.140851 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:34.120500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:41925 (local address 127.26.101.62:42397)
0928 16:46:34.123259 (+ 2759us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:34.123263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:34.123281 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:46:34.124643 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:34.124647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:34.124716 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:34.124822 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:34.126291 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.127068 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.128865 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.129321 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.129424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:34.133512 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:34.133518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:34.133521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:34.133562 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:34.137110 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.138030 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.138035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.138038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.138185 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.138666 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.138670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.138672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.139002 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:34.139276 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:34.140182 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 16:46:34.140456 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2660,"thread_start_us":2456,"threads_started":1}
I20260928 16:46:34.142300 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:34.121237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.3:41925)
0928 16:46:34.122303 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:34.122357 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:34.124379 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:34.125032 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:34.125040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:34.125488 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:34.126105 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:34.126123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.127286 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.127291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:34.128695 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:34.128720 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.129558 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.129564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:34.129814 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:34.130590 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:34.130621 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:34.133189 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:34.137309 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:34.137318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:34.137335 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:34.137881 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:34.138312 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:34.138316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:34.138318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:34.138549 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:34.139252 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:34.139261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:34.139724 (+ 463us) client_negotiation.cc:769] Sending connection context
0928 16:46:34.140074 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 16:46:34.140816 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":450,"threads_started":1}
I20260928 16:46:34.143879 7590 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:34.144264 7590 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:34.145072 7590 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:34.146684 7094 ts_manager.cc:194] Registered new tserver with Master: 3b8439c3e6b04d95b10da263ac8a60fc (127.26.101.3:40819)
I20260928 16:46:34.147791 7094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:41925
I20260928 16:46:34.149096 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:46:34.151068 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7174
Sep 28 16:46:34 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:46:34.174760 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:34.152140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.26.101.1:34011)
0928 16:46:34.153134 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:34.153141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:34.153169 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:46:34.153229 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:34.153235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:34.153543 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:34.153711 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:34.155707 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.157384 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.159981 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.160925 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.161079 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:34.167241 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:34.167264 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:34.167305 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:34.167370 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:34.171292 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.172295 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.172307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.172310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.172519 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.172992 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.172998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.173001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.173510 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:34.173825 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:34.174180 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 16:46:34.174475 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":441,"threads_started":1}
I20260928 16:46:34.176317 7305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:34.262722 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7174
I20260928 16:46:34.277382 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7317
Sep 28 16:46:34 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:46:34.295034 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:34.278371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.26.101.2:40495)
0928 16:46:34.278939 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:34.278949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:34.278984 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:46:34.279055 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:34.279064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:34.279427 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:34.279565 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:34.280937 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.282036 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.283528 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.284213 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.284368 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:34.289090 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:34.289104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:34.289120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:34.289168 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:34.292065 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.292857 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.292865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.292867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.293016 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.293396 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.293399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.293401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.293773 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:34.294060 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:34.294398 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:46:34.294674 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":283,"threads_started":1}
I20260928 16:46:34.296419 7450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:34.379366 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7317
I20260928 16:46:34.398011 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7462
Sep 28 16:46:34 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:46:34.415880 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:34.399097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.26.101.3:40819)
0928 16:46:34.399846 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:34.399853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:34.399884 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:46:34.399947 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:34.399953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:34.400179 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:34.400330 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:34.401796 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.402859 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.404306 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:34.404912 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:34.405034 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:34.409599 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:34.409616 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:34.409632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:34.409694 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:34.412669 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.413535 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.413544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.413546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.413719 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:34.414173 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:34.414179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:34.414183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:34.414638 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:34.414925 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:34.415318 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 16:46:34.415551 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":352,"threads_started":1}
I20260928 16:46:34.417234 7585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:34.501843 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7462
I20260928 16:46:34.516250 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7061
I20260928 16:46:34.517688 7124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:34.625423 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7061
I20260928 16:46:34.645418 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42397
--webserver_interface=127.26.101.62
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:42397
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:34.979470 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:34.979805 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:34.979903 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:34.987852 7603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:34.987989 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:34.988034 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:34.988090 7603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:34.988137 7603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:34.999119 7603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:42397
--ranger_config_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:35.000476 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:35.001821 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:35.014215 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:35.014216 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:35.014245 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:35.015208 7603 server_base.cc:1034] running on GCE node
I20260928 16:46:35.016052 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:35.017557 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:35.018786 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790613995018718 us; error 54 us; skew 500 ppm
I20260928 16:46:35.019214 7603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:35 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613995, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:35.022353 7603 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:35.023949 7603 webserver.cc:460] Webserver started at http://127.26.101.62:42817/ using document root <none> and password file <none>
I20260928 16:46:35.024648 7603 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:35.024782 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:46:35 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613995, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:35.095418 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 16:46:35.098634 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:35.100168 7603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:35.100363 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "cbec47b78a724c5889206903306e222c"
format_stamp: "Formatted at 2026-09-28 16:46:30 on dist-test-slave-t0tw"
server_key: "aa4186bfbb549c8eedaf8228c19aad61"
server_key_iv: "47fec9d8ddea41bbe157575607b05327"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:35.101361 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:35.119860 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:35.124058 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:35.124348 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:35.142561 7603 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:42397
I20260928 16:46:35.142568 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:42397 every 8 connection(s)
I20260928 16:46:35.144097 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:35.150559 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7603
I20260928 16:46:35.151748 7671 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:46:35.151722 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:34011
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:42397
--webserver_port=43063
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:35.156579 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Bootstrap starting.
I20260928 16:46:35.176985 7671 log.cc:824] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:35.186560 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:46:35.187222 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Bootstrap complete.
I20260928 16:46:35.196393 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:35.197311 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec47b78a724c5889206903306e222c, State: Initialized, Role: FOLLOWER
I20260928 16:46:35.197990 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:35.198225 7671 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:35.198342 7671 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:35.198513 7671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:46:35.202638 7671 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:35.203215 7671 leader_election.cc:302] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbec47b78a724c5889206903306e222c; no voters:
I20260928 16:46:35.203835 7671 leader_election.cc:288] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:46:35.203994 7675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:46:35.205224 7675 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [term 2 LEADER]: Becoming Leader. State: Replica: cbec47b78a724c5889206903306e222c, State: Running, Role: LEADER
I20260928 16:46:35.205884 7675 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } }
I20260928 16:46:35.206424 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:35.208731 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cbec47b78a724c5889206903306e222c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } } }
I20260928 16:46:35.209098 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:35.208832 7677 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbec47b78a724c5889206903306e222c. Latest consensus state: current_term: 2 leader_uuid: "cbec47b78a724c5889206903306e222c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbec47b78a724c5889206903306e222c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42397 } } }
I20260928 16:46:35.209199 7677 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:35.223456 7681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:35.229385 7681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:35.230257 7671 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:46:35.231094 7681 catalog_manager.cc:1259] Loaded cluster ID: fcd19216cc0940f1a3545dc28e1b6e09
I20260928 16:46:35.231252 7681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:35.235774 7681 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:35.237605 7681 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Loaded TSK: 0
I20260928 16:46:35.245788 7681 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:46:35.602175 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:35.602806 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:35.603036 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:35.614712 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:35.614954 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:35.615183 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:46:35.631201 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:34011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=43063
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:35.633064 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:35.634981 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:35.684252 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:35.689747 7673 server_base.cc:1034] running on GCE node
W20260928 16:46:35.690752 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:35.691753 7712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:35.692731 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:35.694150 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:35.697594 7673 hybrid_clock.cc:648] HybridClock initialized: now 1790613995697109 us; error 468 us; skew 500 ppm
I20260928 16:46:35.698128 7673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:35 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613995, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:35.711053 7673 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:46:35.734802 7673 webserver.cc:460] Webserver started at http://127.26.101.1:43063/ using document root <none> and password file <none>
I20260928 16:46:35.735853 7673 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:35.736043 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:46:35 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613995, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:35.789405 7673 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.006s sys 0.001s
I20260928 16:46:35.794896 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:35.797119 7673 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:46:35.797417 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "ef16e851d9074f26926ea41f39e86b75"
format_stamp: "Formatted at 2026-09-28 16:46:32 on dist-test-slave-t0tw"
server_key: "11affa906dda5b5942482da39b39e8ce"
server_key_iv: "2cf0e3aa39086e6b5bddc6516f0d8e01"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:35.799053 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:35.819422 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:35.825294 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:35.825623 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:35.827538 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:35.829233 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:35.829332 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:35.829434 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:35.829501 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:35.893536 7673 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:34011
I20260928 16:46:35.893565 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:34011 every 8 connection(s)
I20260928 16:46:35.895543 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:35.906180 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7673
I20260928 16:46:35.907116 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:40495
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:42397
--webserver_port=45777
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:35 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613995, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:35.934141 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:35.899338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.1:59683)
0928 16:46:35.900235 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:35.900288 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:35.902164 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:35.903928 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:35.903941 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:35.904617 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:35.905466 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:35.905499 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:35.915111 (+ 9612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:35.915118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:35.917191 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:35.917217 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:35.919234 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:35.919242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:35.919532 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:35.920364 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:35.920395 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:35.923557 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:35.928138 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:35.928149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:35.928183 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:35.928909 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:35.929560 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:35.929565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:35.929569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:35.929844 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:35.931105 (+ 1261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:35.931115 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:35.931602 (+ 487us) client_negotiation.cc:769] Sending connection context
0928 16:46:35.931934 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 16:46:35.932583 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":426,"threads_started":1}
I20260928 16:46:35.944763 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:35.899063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:59683 (local address 127.26.101.62:42397)
0928 16:46:35.899865 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:35.899889 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:35.900894 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 16:46:35.902835 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:35.902850 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:35.903318 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:35.903735 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:35.910208 (+ 6473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:35.914758 (+ 4550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:35.917467 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:35.918540 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:35.918747 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:35.923976 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:35.923988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:35.924012 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:35.924071 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:35.927898 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:35.929133 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:35.929145 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:35.929154 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:35.929401 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:35.930000 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:35.930005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:35.930009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:35.930779 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:35.930981 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:35.942125 (+ 11144us) server_negotiation.cc:299] Negotiation successful
0928 16:46:35.942696 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":285,"threads_started":1}
I20260928 16:46:35.948690 7834 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:35.949038 7834 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:35.949956 7834 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
W20260928 16:46:35.953639 7834 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:42397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:46:35.953893 7834 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:35.954214 7834 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:35.955822 7834 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:35.956131 7834 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
W20260928 16:46:35.957458 7834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:46:36.387521 7838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:36.387814 7838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:36.387938 7838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:36.399601 7838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:36.399818 7838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:36.400046 7838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:46:36.417357 7838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=45777
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:36.419390 7838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:36.421159 7838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:36.443284 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:36.441181 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:36.452490 7857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:36.453954 7838 server_base.cc:1034] running on GCE node
I20260928 16:46:36.454679 7838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:36.455842 7838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:36.457175 7838 hybrid_clock.cc:648] HybridClock initialized: now 1790613996457101 us; error 62 us; skew 500 ppm
I20260928 16:46:36.457589 7838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:36 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:36.461247 7838 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:46:36.469226 7838 webserver.cc:460] Webserver started at http://127.26.101.2:45777/ using document root <none> and password file <none>
I20260928 16:46:36.470273 7838 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:36.470427 7838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:46:36 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:46:36 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:36.497515 7838 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 16:46:36.507258 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:36.509423 7838 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260928 16:46:36.509748 7838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "99188683cb664e50b6063ade776f31f9"
format_stamp: "Formatted at 2026-09-28 16:46:33 on dist-test-slave-t0tw"
server_key: "3f8e0d07fdeb6e5fe2b4d74b894c2429"
server_key_iv: "0c65c57808c1dd321809a92110b1c7ab"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:36.511644 7838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:36.535216 7838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:36.540025 7838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:36.540338 7838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:36.541541 7838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:36.543028 7838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:36.543149 7838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 16:46:36.543262 7838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:36.543334 7838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:36.613088 7838 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:40495
I20260928 16:46:36.613111 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:40495 every 8 connection(s)
I20260928 16:46:36.615033 7838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:36.624320 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7838
I20260928 16:46:36.625126 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.26.101.20:56059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:40819
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:42397
--webserver_port=37439
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:36 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:36.689805 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:36.620208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:53163 (local address 127.26.101.62:42397)
0928 16:46:36.648544 (+ 28336us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:36.648551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:36.648579 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:46:36.650926 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:36.650932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:36.651014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:36.651182 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:36.653514 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:36.654638 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:36.657157 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:36.658030 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:36.658158 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:36.670860 (+ 12702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:36.670870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:36.670875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:36.670930 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:36.677840 (+ 6910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:36.679754 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:36.679762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:36.679766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:36.679938 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:36.687254 (+ 7316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:36.687262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:36.687267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:36.687721 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:36.687883 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:36.689111 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0928 16:46:36.689411 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28191,"thread_start_us":28069,"threads_started":1}
I20260928 16:46:36.692515 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:36.619478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.2:53163)
0928 16:46:36.646352 (+ 26874us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:36.646988 (+ 636us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:36.650476 (+ 3488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:36.651518 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:36.651535 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:36.652298 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:36.653247 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:36.653279 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:36.654862 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:36.654868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:36.656943 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:36.656962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:36.658612 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:36.658619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:36.658924 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:36.660336 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:36.660407 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:36.670455 (+ 10048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:36.678729 (+ 8274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:36.678741 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:36.678765 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:36.679569 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:36.680101 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:36.680105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:36.680109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:36.683178 (+ 3069us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:36.688007 (+ 4829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:36.688017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:36.688526 (+ 509us) client_negotiation.cc:769] Sending connection context
0928 16:46:36.690429 (+ 1903us) client_negotiation.cc:240] Negotiation successful
0928 16:46:36.691041 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26478,"thread_start_us":26201,"threads_started":1}
I20260928 16:46:36.695477 7980 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:36.695850 7980 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:36.696883 7980 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:36.699306 7636 ts_manager.cc:194] Registered new tserver with Master: 99188683cb664e50b6063ade776f31f9 (127.26.101.2:40495)
I20260928 16:46:36.701582 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:53163
Sep 28 16:46:36 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:36.960932 7834 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:36.961432 7834 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:36.963040 7636 ts_manager.cc:194] Registered new tserver with Master: ef16e851d9074f26926ea41f39e86b75 (127.26.101.1:34011)
I20260928 16:46:36.964293 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:59683
W20260928 16:46:37.164213 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:37.164633 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:37.164783 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:37.173066 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:37.173209 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:37.173386 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:46:37.184744 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.26.101.20:56059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:40819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=37439
--tserver_master_addrs=127.26.101.62:42397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:37.186297 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:37.187777 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:37.203464 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:37.204475 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:37.236220 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:37.236485 7984 server_base.cc:1034] running on GCE node
I20260928 16:46:37.237259 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:37.238404 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:37.239730 7984 hybrid_clock.cc:648] HybridClock initialized: now 1790613997239654 us; error 68 us; skew 500 ppm
I20260928 16:46:37.240106 7984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:37 dist-test-slave-t0tw krb5kdc[6613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613997, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:37.243849 7984 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:46:37.245945 7984 webserver.cc:460] Webserver started at http://127.26.101.3:37439/ using document root <none> and password file <none>
I20260928 16:46:37.246723 7984 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:37.246870 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:46:37 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613997, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
Sep 28 16:46:37 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613996, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for HTTP/127.26.101.20@KRBTEST.COM
I20260928 16:46:37.317695 7984 fs_manager.cc:709] Time spent opening directory manager: real 0.041s user 0.027s sys 0.011s
I20260928 16:46:37.330698 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:37.332998 7984 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.010s
I20260928 16:46:37.333220 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "3b8439c3e6b04d95b10da263ac8a60fc"
format_stamp: "Formatted at 2026-09-28 16:46:34 on dist-test-slave-t0tw"
server_key: "3d2305bfd9fb3e423f0da6a1a838d495"
server_key_iv: "31bba40a9a2ae016b93f45dde8b0f0b3"
server_key_version: "kuduclusterkey@0"
I20260928 16:46:37.334591 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:37.428277 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:37.434273 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:37.434659 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:37.446550 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:37.460120 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:37.460342 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:46:37.460505 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:37.460613 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:37.642769 7984 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:40819
I20260928 16:46:37.642969 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:40819 every 8 connection(s)
I20260928 16:46:37.644901 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:37.653445 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 7984
Sep 28 16:46:37 dist-test-slave-t0tw krb5kdc[6613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613997, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:37.671823 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.648128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:45841 (local address 127.26.101.62:42397)
0928 16:46:37.648708 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.648713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.649723 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 16:46:37.651963 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.651970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.652054 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.652154 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:37.654172 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.655524 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.657990 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.658727 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.658835 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.663906 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:37.663913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:37.663918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:37.663969 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:37.668042 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.669075 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.669081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.669084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.669267 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.669714 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.669720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.669723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.670112 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:37.670244 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.671304 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.671486 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":329,"threads_started":1}
I20260928 16:46:37.673380 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.648722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:42397 (local address 127.26.101.3:45841)
0928 16:46:37.649557 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.649596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.651653 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.652431 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.652440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.652948 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:37.653928 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.653961 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.655741 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.655747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.657803 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.657820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.659294 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.659302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.659592 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.660437 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:37.660473 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:37.663603 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:37.668227 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:37.668238 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:37.668260 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:37.668946 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:37.669386 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:37.669390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:37.669392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:37.669596 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:37.670283 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:37.670293 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:37.670786 (+ 493us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.671210 (+ 424us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.671845 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":379,"threads_started":1}
I20260928 16:46:37.674969 8122 heartbeater.cc:347] Connected to a master server at 127.26.101.62:42397
I20260928 16:46:37.675315 8122 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:37.676067 8122 heartbeater.cc:511] Master 127.26.101.62:42397 requested a full tablet report, sending...
I20260928 16:46:37.677682 7636 ts_manager.cc:194] Registered new tserver with Master: 3b8439c3e6b04d95b10da263ac8a60fc (127.26.101.3:40819)
I20260928 16:46:37.678776 7636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:45841
I20260928 16:46:37.678980 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.654546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.26.101.62:42397)
0928 16:46:37.663562 (+ 9016us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.663569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.664241 (+ 672us) server_negotiation.cc:407] Connection header received
0928 16:46:37.664460 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.664465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.664548 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.664655 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:37.666425 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.667400 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.669075 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.669853 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.669980 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.673388 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:37.673393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:37.673397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:37.673432 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:37.676225 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.676907 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.676912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.676916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.677075 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.677542 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.677547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.677549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.677889 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:37.678135 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.678463 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.678629 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8854,"thread_start_us":9874,"threads_started":1}
I20260928 16:46:37.681658 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:46:37.697566 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.684883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.26.101.62:42397)
0928 16:46:37.685052 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.685057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.685392 (+ 335us) server_negotiation.cc:407] Connection header received
0928 16:46:37.685533 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.685536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.685601 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.685682 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:37.687320 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.688024 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.689424 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.689850 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.689948 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.692659 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:37.692664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:37.692668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:37.692704 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:37.695214 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.695923 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.695926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.695929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.696060 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:37.696454 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:37.696457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:37.696459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:37.696714 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:37.696898 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.697132 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.697318 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:46:37.707119 7980 heartbeater.cc:503] Master 127.26.101.62:42397 was elected leader, sending a full tablet report...
I20260928 16:46:37.703676 7636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:46:37.712256 7636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:46:37.744606 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.727875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40495 (local address 127.0.0.1:54906)
0928 16:46:37.729103 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.729148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.729323 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.732002 (+ 2679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.732011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.732047 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:37.732543 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.732558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.736258 (+ 3700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.736265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.740769 (+ 4504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.740794 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.743134 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.743142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.743413 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.743482 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.743625 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.743867 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"threads_started":1}
I20260928 16:46:37.746927 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.728396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.26.101.2:40495)
0928 16:46:37.729939 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.729949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.729979 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:46:37.730084 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.730091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.730383 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.732782 (+ 2399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:37.733066 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.736083 (+ 3017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.741066 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.742790 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.745931 (+ 3141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.746254 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.746340 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.746547 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1369,"thread_start_us":1191,"threads_started":1}
I20260928 16:46:37.754408 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.728021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:34011 (local address 127.0.0.1:59284)
0928 16:46:37.745557 (+ 17536us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.745579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.745731 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.748080 (+ 2349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.748087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.748130 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:37.748440 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.748454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.750383 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.750389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.752182 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.752196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.753547 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.753553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.753733 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.753776 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.753866 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.754014 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17396,"thread_start_us":675,"threads_started":1}
I20260928 16:46:37.754405 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.728350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.26.101.1:34011)
0928 16:46:37.744368 (+ 16018us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.744377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.747308 (+ 2931us) server_negotiation.cc:407] Connection header received
0928 16:46:37.747463 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.747473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.747798 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.748050 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:37.748631 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.750185 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.752344 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.753366 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.753498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.753769 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.753925 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.754092 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15837,"thread_start_us":15658,"threads_started":1}
I20260928 16:46:37.758576 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.730053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40819 (local address 127.0.0.1:42464)
0928 16:46:37.749768 (+ 19715us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.749791 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.749955 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.751176 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.751182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.751214 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:37.751690 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.751703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.753976 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.753982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.756426 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.756441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.757801 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.757807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.757979 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.758010 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.758087 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.758228 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19554,"thread_start_us":19386,"threads_started":1}
I20260928 16:46:37.759495 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.730599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.26.101.3:40819)
0928 16:46:37.749242 (+ 18643us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.749252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.750100 (+ 848us) server_negotiation.cc:407] Connection header received
0928 16:46:37.750235 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.750245 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.750597 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.750832 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:37.752006 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.753781 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.756592 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.757574 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.757713 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.757943 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.758937 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.759167 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18428,"thread_start_us":18230,"threads_started":1}
I20260928 16:46:37.758899 7769 tablet_service.cc:1467] Processing CreateTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 (DEFAULT_TABLE table=test-table [id=547e9ab5c1914383812fb888167fd31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:37.758899 7915 tablet_service.cc:1467] Processing CreateTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 (DEFAULT_TABLE table=test-table [id=547e9ab5c1914383812fb888167fd31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:37.760538 7769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a97a3d8486946a7ad1ddec285e27a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:37.760543 7915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a97a3d8486946a7ad1ddec285e27a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:37.762755 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 (DEFAULT_TABLE table=test-table [id=547e9ab5c1914383812fb888167fd31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:37.764197 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a97a3d8486946a7ad1ddec285e27a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:37.776057 8144 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Bootstrap starting.
I20260928 16:46:37.779946 8144 tablet_bootstrap.cc:654] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:37.781041 8144 log.cc:824] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:37.782742 8146 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Bootstrap starting.
I20260928 16:46:37.783033 8145 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Bootstrap starting.
I20260928 16:46:37.786002 8144 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: No bootstrap required, opened a new log
I20260928 16:46:37.786319 8144 ts_tablet_manager.cc:1397] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 16:46:37.786576 8146 tablet_bootstrap.cc:654] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:37.786669 8145 tablet_bootstrap.cc:654] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:37.787707 8146 log.cc:824] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:37.787977 8145 log.cc:824] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:37.790970 8145 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: No bootstrap required, opened a new log
I20260928 16:46:37.791311 8146 tablet_bootstrap.cc:492] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: No bootstrap required, opened a new log
I20260928 16:46:37.791320 8145 ts_tablet_manager.cc:1397] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 16:46:37.791597 8146 ts_tablet_manager.cc:1397] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 16:46:37.797381 8144 raft_consensus.cc:348] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.797920 8144 raft_consensus.cc:374] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:37.798087 8144 raft_consensus.cc:729] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8439c3e6b04d95b10da263ac8a60fc, State: Initialized, Role: FOLLOWER
I20260928 16:46:37.798833 8144 consensus_queue.cc:260] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.799347 8145 raft_consensus.cc:348] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.799872 8145 raft_consensus.cc:374] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:37.800019 8145 raft_consensus.cc:729] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99188683cb664e50b6063ade776f31f9, State: Initialized, Role: FOLLOWER
I20260928 16:46:37.800163 8122 heartbeater.cc:503] Master 127.26.101.62:42397 was elected leader, sending a full tablet report...
I20260928 16:46:37.800927 8144 ts_tablet_manager.cc:1428] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260928 16:46:37.800853 8145 consensus_queue.cc:260] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.801898 8146 raft_consensus.cc:348] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.802405 8146 raft_consensus.cc:374] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:37.802526 8146 raft_consensus.cc:729] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef16e851d9074f26926ea41f39e86b75, State: Initialized, Role: FOLLOWER
I20260928 16:46:37.802757 8145 ts_tablet_manager.cc:1428] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 16:46:37.803241 8146 consensus_queue.cc:260] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.804417 7834 heartbeater.cc:503] Master 127.26.101.62:42397 was elected leader, sending a full tablet report...
I20260928 16:46:37.805190 8146 ts_tablet_manager.cc:1428] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260928 16:46:37.884126 7981 tablet.cc:2369] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:37.894224 8152 raft_consensus.cc:482] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:46:37.894550 8152 raft_consensus.cc:504] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.896420 8152 leader_election.cc:288] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99188683cb664e50b6063ade776f31f9 (127.26.101.2:40495), 3b8439c3e6b04d95b10da263ac8a60fc (127.26.101.3:40819)
W20260928 16:46:37.897610 8123 tablet.cc:2369] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:46:37.898820 7835 tablet.cc:2369] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:37.905409 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.897245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:38111 (local address 127.26.101.2:40495)
0928 16:46:37.897433 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.897438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.898048 (+ 610us) server_negotiation.cc:407] Connection header received
0928 16:46:37.898360 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.898365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.898446 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.898506 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:37.899393 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.900486 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.903181 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.904302 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.904505 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.904591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.905002 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.905166 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:46:37.905457 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.896684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:40495 (local address 127.26.101.1:38111)
0928 16:46:37.897929 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.897949 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.898168 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.898701 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.898705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.898736 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:37.899209 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.899223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.900678 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.900683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.902919 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.902935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.904581 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.904588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.904812 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.904840 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.904946 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.905091 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":6541,"threads_started":1}
I20260928 16:46:37.906658 7935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a97a3d8486946a7ad1ddec285e27a38" candidate_uuid: "ef16e851d9074f26926ea41f39e86b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99188683cb664e50b6063ade776f31f9" is_pre_election: true
I20260928 16:46:37.907251 7935 raft_consensus.cc:2454] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef16e851d9074f26926ea41f39e86b75 in term 0.
I20260928 16:46:37.908013 7723 leader_election.cc:302] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99188683cb664e50b6063ade776f31f9, ef16e851d9074f26926ea41f39e86b75; no voters:
I20260928 16:46:37.908571 8152 raft_consensus.cc:2781] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:46:37.908720 8152 raft_consensus.cc:482] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:46:37.908819 8152 raft_consensus.cc:3037] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:37.909924 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.897185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:46351 (local address 127.26.101.3:40819)
0928 16:46:37.897398 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.897404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.903724 (+ 6320us) server_negotiation.cc:407] Connection header received
0928 16:46:37.903831 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.903837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.903924 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.904037 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:37.904768 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.905616 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.907785 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.908968 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.909127 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.909188 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.909608 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.909722 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:46:37.910079 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.896874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40819 (local address 127.26.101.1:46351)
0928 16:46:37.903557 (+ 6683us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:37.903588 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:37.903710 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:37.904052 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:37.904057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:37.904085 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:37.904532 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.904544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.905786 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.905791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.907573 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:37.907590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.909209 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.909215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:37.909389 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.909424 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:46:37.909537 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:46:37.909692 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6536,"thread_start_us":701,"threads_started":1}
I20260928 16:46:37.911141 8072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a97a3d8486946a7ad1ddec285e27a38" candidate_uuid: "ef16e851d9074f26926ea41f39e86b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" is_pre_election: true
I20260928 16:46:37.911657 8072 raft_consensus.cc:2454] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef16e851d9074f26926ea41f39e86b75 in term 0.
I20260928 16:46:37.911638 8152 raft_consensus.cc:504] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.912269 8152 leader_election.cc:288] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [CANDIDATE]: Term 1 election: Requested vote from peers 99188683cb664e50b6063ade776f31f9 (127.26.101.2:40495), 3b8439c3e6b04d95b10da263ac8a60fc (127.26.101.3:40819)
I20260928 16:46:37.912796 7935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a97a3d8486946a7ad1ddec285e27a38" candidate_uuid: "ef16e851d9074f26926ea41f39e86b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99188683cb664e50b6063ade776f31f9"
I20260928 16:46:37.912902 8072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a97a3d8486946a7ad1ddec285e27a38" candidate_uuid: "ef16e851d9074f26926ea41f39e86b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8439c3e6b04d95b10da263ac8a60fc"
I20260928 16:46:37.913007 7935 raft_consensus.cc:3037] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:37.913087 8072 raft_consensus.cc:3037] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:37.915436 7935 raft_consensus.cc:2454] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef16e851d9074f26926ea41f39e86b75 in term 1.
I20260928 16:46:37.915508 8072 raft_consensus.cc:2454] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef16e851d9074f26926ea41f39e86b75 in term 1.
I20260928 16:46:37.915912 7723 leader_election.cc:302] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99188683cb664e50b6063ade776f31f9, ef16e851d9074f26926ea41f39e86b75; no voters:
I20260928 16:46:37.916251 8152 raft_consensus.cc:2781] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:37.916706 8152 raft_consensus.cc:686] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 1 LEADER]: Becoming Leader. State: Replica: ef16e851d9074f26926ea41f39e86b75, State: Running, Role: LEADER
I20260928 16:46:37.917291 8152 consensus_queue.cc:237] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } }
I20260928 16:46:37.921732 7634 catalog_manager.cc:5495] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef16e851d9074f26926ea41f39e86b75 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "ef16e851d9074f26926ea41f39e86b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef16e851d9074f26926ea41f39e86b75" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 34011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 } health_report { overall_health: UNKNOWN } } }
I20260928 16:46:37.980249 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:37.974622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.26.101.1:34011)
0928 16:46:37.974870 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:37.974875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:37.974894 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:46:37.975027 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:37.975030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:37.975151 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:37.975287 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:46:37.975702 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.976415 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.977911 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:37.978328 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:37.978420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:37.978996 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:46:37.979600 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:46:37.979722 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:37.979883 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 16:46:37.979964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 16:46:37.989073 7935 raft_consensus.cc:1261] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 1 FOLLOWER]: Refusing update from remote peer ef16e851d9074f26926ea41f39e86b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:37.989087 8072 raft_consensus.cc:1261] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 1 FOLLOWER]: Refusing update from remote peer ef16e851d9074f26926ea41f39e86b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:37.990000 8155 consensus_queue.cc:1035] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8439c3e6b04d95b10da263ac8a60fc" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:37.990409 8152 consensus_queue.cc:1035] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99188683cb664e50b6063ade776f31f9" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:38.007359 8158 mvcc.cc:205] Tried to move back new op lower bound from 7334354935752773632 to 7334354935473205248. Current Snapshot: MvccSnapshot[applied={T|T < 7334354935752773632}]
I20260928 16:46:38.009701 8159 mvcc.cc:205] Tried to move back new op lower bound from 7334354935752773632 to 7334354935473205248. Current Snapshot: MvccSnapshot[applied={T|T < 7334354935752773632}]
I20260928 16:46:38.021672 7635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:38.022046 7635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:38.026556 7635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbec47b78a724c5889206903306e222c: Sending DeleteTablet for 3 replicas of tablet 6a97a3d8486946a7ad1ddec285e27a38
I20260928 16:46:38.027988 7769 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:38 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:59284
I20260928 16:46:38.027982 7915 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:38 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:54906
I20260928 16:46:38.028745 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a97a3d8486946a7ad1ddec285e27a38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:38 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:42464
I20260928 16:46:38.037496 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7673
I20260928 16:46:38.039876 8169 tablet_replica.cc:330] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: stopping tablet replica
I20260928 16:46:38.041038 8169 raft_consensus.cc:2229] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:38.041692 8169 raft_consensus.cc:2258] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:38.044406 8169 ts_tablet_manager.cc:1905] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:38.047258 8167 tablet_replica.cc:330] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: stopping tablet replica
I20260928 16:46:38.047789 8167 raft_consensus.cc:2229] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:46:38.048542 8167 raft_consensus.cc:2258] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:38.049443 8168 tablet_replica.cc:330] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: stopping tablet replica
I20260928 16:46:38.049925 8168 raft_consensus.cc:2229] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:38.050500 8168 raft_consensus.cc:2258] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:38.053129 8168 ts_tablet_manager.cc:1905] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:38.054824 8167 ts_tablet_manager.cc:1905] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:38.058158 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:38.038491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.26.101.1:34011)
0928 16:46:38.039704 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:38.039710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:38.039733 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:46:38.039802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:38.039806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:38.039887 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:38.040001 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:38.042159 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.043553 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.045736 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.046525 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.046655 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:38.050458 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:38.050476 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:38.050501 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:38.050553 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:38.054906 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.055755 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.055768 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.055771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.055959 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.056395 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.056400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.056404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.056908 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:38.057356 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:38.057532 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 16:46:38.057767 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076}
I20260928 16:46:38.059295 8169 ts_tablet_manager.cc:1918] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:38.059338 7829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:38.059497 8169 log.cc:1196] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/6a97a3d8486946a7ad1ddec285e27a38
I20260928 16:46:38.060083 8169 ts_tablet_manager.cc:1939] T 6a97a3d8486946a7ad1ddec285e27a38 P 3b8439c3e6b04d95b10da263ac8a60fc: Deleting consensus metadata
I20260928 16:46:38.062522 7620 catalog_manager.cc:4841] TS 3b8439c3e6b04d95b10da263ac8a60fc (127.26.101.3:40819): tablet 6a97a3d8486946a7ad1ddec285e27a38 (table test-table [id=547e9ab5c1914383812fb888167fd31b]) successfully deleted
I20260928 16:46:38.063009 8168 ts_tablet_manager.cc:1918] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:38.063233 8168 log.cc:1196] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/6a97a3d8486946a7ad1ddec285e27a38
I20260928 16:46:38.063901 8168 ts_tablet_manager.cc:1939] T 6a97a3d8486946a7ad1ddec285e27a38 P 99188683cb664e50b6063ade776f31f9: Deleting consensus metadata
I20260928 16:46:38.065531 7622 catalog_manager.cc:4841] TS 99188683cb664e50b6063ade776f31f9 (127.26.101.2:40495): tablet 6a97a3d8486946a7ad1ddec285e27a38 (table test-table [id=547e9ab5c1914383812fb888167fd31b]) successfully deleted
I20260928 16:46:38.164002 8167 ts_tablet_manager.cc:1918] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:38.164294 8167 log.cc:1196] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/6a97a3d8486946a7ad1ddec285e27a38
I20260928 16:46:38.164858 8167 ts_tablet_manager.cc:1939] T 6a97a3d8486946a7ad1ddec285e27a38 P ef16e851d9074f26926ea41f39e86b75: Deleting consensus metadata
I20260928 16:46:38.166658 7623 catalog_manager.cc:4841] TS ef16e851d9074f26926ea41f39e86b75 (127.26.101.1:34011): tablet 6a97a3d8486946a7ad1ddec285e27a38 (table test-table [id=547e9ab5c1914383812fb888167fd31b]) successfully deleted
I20260928 16:46:38.167259 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7673
I20260928 16:46:38.187714 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7838
I20260928 16:46:38.203414 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:38.188688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.26.101.2:40495)
0928 16:46:38.188860 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:38.188865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:38.188885 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:46:38.188943 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:38.188946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:38.189012 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:38.189095 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:38.190491 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.191380 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.192919 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.193360 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.193476 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:38.196342 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:38.196367 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:38.196389 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:38.196442 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:38.199917 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.200863 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.200871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.200874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.201046 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.201527 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.201531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.201534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.201965 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:38.202232 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:38.202702 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 16:46:38.202936 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:46:38.204355 7975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:38.294162 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7838
I20260928 16:46:38.309633 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7984
I20260928 16:46:38.323874 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:38.310427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.26.101.3:40819)
0928 16:46:38.310614 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:38.310618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:38.310639 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:46:38.310722 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:38.310726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:38.310808 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:38.310899 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:38.312430 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.313256 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.314634 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:38.315277 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:38.315385 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:38.317920 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:38.317936 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:38.317952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:38.317988 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:38.320891 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.321726 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.321734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.321736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.321908 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:38.322359 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:38.322365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:38.322368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:38.322769 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:38.323003 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:38.323380 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 16:46:38.323598 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:46:38.324558 8117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:38.413223 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7984
I20260928 16:46:38.428596 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 7603
I20260928 16:46:38.429952 7666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:38.543470 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 7603
I20260928 16:46:38.565845 27028 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:46:38.902223 27028 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:46:38.902459 27028 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:46:39.248916 27028 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:46:39.249 UTC [6643] LOG: received smart shutdown request
2026-09-28 16:46:39.252 UTC [6643] LOG: background worker "logical replication launcher" (PID 6652) exited with exit code 1
2026-09-28 16:46:39.252 UTC [6647] LOG: shutting down
2026-09-28 16:46:39.439 UTC [6643] LOG: database system is shut down
2026-09-28T16:46:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54451 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:46:39 dist-test-slave-t0tw krb5kdc[8182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:46:39 dist-test-slave-t0tw krb5kdc[8182](info): set up 2 sockets
Sep 28 16:46:39 dist-test-slave-t0tw krb5kdc[8182](info): commencing operation
krb5kdc: starting...
W20260928 16:46:40.571277 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:46:40 dist-test-slave-t0tw krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:46:40.826325 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46017
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:41.163411 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:41.163739 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:41.163877 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:41.171386 8195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:41.171507 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:41.171581 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:41.171675 8195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:41.171730 8195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:41.182085 8195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:41.183597 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:41.185040 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:41.197309 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:41.197335 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:41.197288 8200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:41.198246 8195 server_base.cc:1034] running on GCE node
I20260928 16:46:41.199131 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:41.200730 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:41.201956 8195 hybrid_clock.cc:648] HybridClock initialized: now 1790614001201903 us; error 39 us; skew 500 ppm
I20260928 16:46:41.202313 8195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:41.208954 8195 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:41.210600 8195 webserver.cc:460] Webserver started at http://127.26.101.62:43177/ using document root <none> and password file <none>
I20260928 16:46:41.211373 8195 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:41.211478 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:41.211755 8195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:41.214805 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "1ee8c6848af443399ad499c7c6d50ce9"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "98e701bfd580d25f33fbc36299fca012"
server_key_iv: "33a4ffd00964e56828621f4435da06f9"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.215673 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "1ee8c6848af443399ad499c7c6d50ce9"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "98e701bfd580d25f33fbc36299fca012"
server_key_iv: "33a4ffd00964e56828621f4435da06f9"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.220872 8195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 16:46:41.224757 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:41.226385 8195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:46:41.226568 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "1ee8c6848af443399ad499c7c6d50ce9"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "98e701bfd580d25f33fbc36299fca012"
server_key_iv: "33a4ffd00964e56828621f4435da06f9"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.226778 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:41.242165 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:41.246481 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:41.246773 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:41.264932 8195 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:46017
I20260928 16:46:41.264964 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:46017 every 8 connection(s)
I20260928 16:46:41.266368 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:41.269528 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8195
I20260928 16:46:41.269773 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:46:41.270627 27028 external_mini_cluster.cc:1428] Setting key b2cd2b95ffaaf87519d1e948b3d68a38
I20260928 16:46:41.270689 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:41.280020 8263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.26.101.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.61@KRBTEST.COM" created.
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:41.304634 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.283468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.26.101.62:46017)
0928 16:46:41.285231 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:41.285239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:41.285292 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:46:41.286812 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:41.286818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:41.287123 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:41.287323 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:41.288648 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.289965 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.291388 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.291955 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.292144 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.296585 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:41.296596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:41.296615 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:41.296656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:41.300297 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.301094 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.301111 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.301114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.301303 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.301693 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.301698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.301701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.302399 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:41.302599 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:41.303178 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 16:46:41.303590 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":1539,"spinlock_wait_cycles":17152,"thread_start_us":1302,"threads_started":1}
I20260928 16:46:41.304692 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.283468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46017 (local address 127.0.0.1:45638)
0928 16:46:41.284372 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.284420 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:41.286435 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:41.287367 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:41.287374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:41.288100 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:41.288447 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.288464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.290110 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.290114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.291260 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.291273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.292196 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.292204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.292436 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.293077 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:41.293101 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:41.296311 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:41.300461 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.300473 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.300484 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.300938 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.301413 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.301417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.301419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.301586 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.302531 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:41.302538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:41.302744 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 16:46:41.302952 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 16:46:41.303450 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":50,"thread_start_us":386,"threads_started":1}
I20260928 16:46:41.307942 8263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:46:41.309283 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.308729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56726)
0928 16:46:41.308957 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.309145 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 16:46:41.309616 8211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
I20260928 16:46:41.311272 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.310829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56728)
0928 16:46:41.310984 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.311059 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:46:41.311478 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:41265: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
I20260928 16:46:41.326599 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:41265
--webserver_interface=127.26.101.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:41.360822 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } attempt: 1
I20260928 16:46:41.362090 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.361583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56732)
0928 16:46:41.361810 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.361940 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:46:41.363193 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.362754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56740)
0928 16:46:41.362894 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.362978 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:46:41.363389 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:41265: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
I20260928 16:46:41.431615 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } attempt: 2
I20260928 16:46:41.432883 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.432372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56748)
0928 16:46:41.432625 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.432736 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:46:41.433974 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.433601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56754)
0928 16:46:41.433749 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.433836 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:46:41.434186 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:41265: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
I20260928 16:46:41.525441 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } attempt: 3
I20260928 16:46:41.526635 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.526200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56762)
0928 16:46:41.526395 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.526503 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:46:41.527711 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.527337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56778)
0928 16:46:41.527476 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.527551 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:46:41.527908 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:41265: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
W20260928 16:46:41.654354 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:41.654685 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:41.654805 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:41.662428 8270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:41.662560 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:41.662626 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:41.662709 8270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:41.662773 8270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:41.671182 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } attempt: 4
I20260928 16:46:41.672487 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.671887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56782)
0928 16:46:41.672251 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.672349 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260928 16:46:41.673503 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.673149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56798)
0928 16:46:41.673287 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.673379 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:46:41.673517 8270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.101.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
W20260928 16:46:41.673693 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:41265: Network error: Client connection negotiation failed: client connection to 127.26.101.61:41265: connect: Connection refused (error 111)
I20260928 16:46:41.674925 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:41.676291 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:41.688700 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:41.688751 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:41.688675 8278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:41.689779 8270 server_base.cc:1034] running on GCE node
I20260928 16:46:41.690608 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:41.692379 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:41.693667 8270 hybrid_clock.cc:648] HybridClock initialized: now 1790614001693596 us; error 61 us; skew 500 ppm
I20260928 16:46:41.694067 8270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:41.697554 8270 init.cc:377] Logged in from keytab as kudu/127.26.101.61@KRBTEST.COM (short username kudu)
I20260928 16:46:41.699258 8270 webserver.cc:460] Webserver started at http://127.26.101.61:42185/ using document root <none> and password file <none>
I20260928 16:46:41.700003 8270 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:41.700130 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:41.700438 8270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:41.704066 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data/instance:
uuid: "d0de179e2c784be38f65fa2f8ff11cf1"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "7326b496c24dd30ec6d7dbebaa182285"
server_key_iv: "4bc42cd8c771ee057307e4ec782c7865"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.704928 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal/instance:
uuid: "d0de179e2c784be38f65fa2f8ff11cf1"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "7326b496c24dd30ec6d7dbebaa182285"
server_key_iv: "4bc42cd8c771ee057307e4ec782c7865"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.710515 8270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 16:46:41.714277 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:41.715758 8270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:41.715939 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "d0de179e2c784be38f65fa2f8ff11cf1"
format_stamp: "Formatted at 2026-09-28 16:46:41 on dist-test-slave-t0tw"
server_key: "7326b496c24dd30ec6d7dbebaa182285"
server_key_iv: "4bc42cd8c771ee057307e4ec782c7865"
server_key_version: "encryptionkey@0"
I20260928 16:46:41.716144 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:41.739364 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:41.743953 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:41.744244 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:41.762501 8270 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.61:41265
I20260928 16:46:41.762537 8336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.61:41265 every 8 connection(s)
I20260928 16:46:41.763921 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
I20260928 16:46:41.768040 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:41.768940 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8270
I20260928 16:46:41.769138 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-1/wal/instance
I20260928 16:46:41.769699 27028 external_mini_cluster.cc:1428] Setting key 590c9ebce867f924ecfdf1c1803208af
I20260928 16:46:41.777344 8337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.60@KRBTEST.COM" created.
I20260928 16:46:41.799326 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.780781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.26.101.62:46017)
0928 16:46:41.780969 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:41.780973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:41.782491 (+ 1518us) server_negotiation.cc:407] Connection header received
0928 16:46:41.784533 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:41.784539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:41.784622 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:41.784712 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:41.786379 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.787053 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.788827 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.789511 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.789674 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.793367 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:41.793373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:41.793377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:41.793414 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:41.796042 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.796853 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.796859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.796863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.796992 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.797441 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.797446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.797449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.797756 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:41.797907 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:41.798686 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 16:46:41.798947 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:46:41.800387 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.781431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46017 (local address 127.0.0.1:45648)
0928 16:46:41.782288 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.782346 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:41.784221 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:41.784991 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:41.785000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:41.785560 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:41.786180 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.786200 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.787263 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.787267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.788624 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.788643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.789772 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.789778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.790004 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.790717 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:41.790740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:41.793097 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:41.796215 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.796225 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.796245 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.796728 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.797106 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.797112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.797116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.797324 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.797896 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:41.797905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:41.798307 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 16:46:41.798575 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 16:46:41.799191 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":337,"threads_started":1}
I20260928 16:46:41.802386 8337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:46:41.817087 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.803234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56804)
0928 16:46:41.803426 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.803446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:41.803534 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:41.804573 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:41.804578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:41.805005 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:41.805146 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.805155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.806511 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.806516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.807430 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.807443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.808291 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.808297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.808459 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.809022 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:41.809040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:41.811302 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:41.814569 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.814575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.814579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.814944 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.815351 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.815356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.815360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.815495 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.816134 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:41.816139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:41.816288 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 16:46:41.816401 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:46:41.816687 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:46:41.817165 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.803493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.26.101.61:41265)
0928 16:46:41.804161 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:41.804167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:41.804192 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:46:41.804236 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:41.804242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:41.804429 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:41.804557 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:41.805295 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.806374 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.807558 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.808122 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.808233 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.811497 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:41.811509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:41.811525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:41.811558 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:41.814434 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.815111 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.815120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.815124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.815246 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.815608 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.815614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.815616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.816014 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:41.816217 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:41.816579 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:46:41.816805 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":392,"threads_started":1}
I20260928 16:46:41.819888 8337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:46:41.821162 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.820631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60862)
0928 16:46:41.820828 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.821001 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 16:46:41.821473 8287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
I20260928 16:46:41.823318 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.822816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60878)
0928 16:46:41.822995 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.823189 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 16:46:41.823618 8337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:41.824682 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:45257
--webserver_interface=127.26.101.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:41.873039 8337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 1
I20260928 16:46:41.874436 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.873831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60892)
0928 16:46:41.874082 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.874225 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 16:46:41.875545 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.875145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60906)
0928 16:46:41.875312 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.875404 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:46:41.875792 8337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:41.944046 8337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 2
I20260928 16:46:41.945434 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.944853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60914)
0928 16:46:41.945097 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.945221 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 16:46:41.946543 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.946129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60916)
0928 16:46:41.946301 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.946392 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:46:41.946784 8337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:41.972961 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } attempt: 5
Sep 28 16:46:41 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:46:41.986444 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.973748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56818)
0928 16:46:41.973963 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.973984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:41.974130 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:41.974503 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:41.974508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:41.974914 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:41.975131 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.975141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.976135 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.976141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.977011 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:41.977023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.977754 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.977760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:41.977935 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.978573 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:41.978591 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:41.980824 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:41.984012 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.984019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.984024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.984417 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.984799 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:41.984803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:41.984805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:41.984942 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:41.985476 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:41.985480 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:41.985638 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 16:46:41.985771 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:46:41.986074 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:46:41.986447 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.973800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.26.101.61:41265)
0928 16:46:41.973988 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:41.973993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:41.974069 (+ 76us) server_negotiation.cc:407] Connection header received
0928 16:46:41.974276 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:41.974280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:41.974362 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:41.974449 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:41.975307 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.975996 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.977159 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:41.977596 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:41.977708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:41.981030 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:41.981035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:41.981038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:41.981072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:41.983850 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.984538 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.984543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.984546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.984670 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:41.985049 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:41.985053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:41.985056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:41.985355 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:41.985490 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:41.985900 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 16:46:41.986134 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:46:41.987665 8263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:46:41.988752 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.988344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60930)
0928 16:46:41.988518 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.988600 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 16:46:41.989703 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:41.989373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60938)
0928 16:46:41.989512 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:41.989598 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:46:41.989897 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:42.038043 8337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 3
I20260928 16:46:42.039414 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.038756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60954)
0928 16:46:42.039018 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.039196 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 16:46:42.040531 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.040040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60966)
0928 16:46:42.040294 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.040387 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260928 16:46:42.040764 8337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:42.041092 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 1
I20260928 16:46:42.041983 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.041572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60970)
0928 16:46:42.041751 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.041841 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:46:42.042989 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.042657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60984)
0928 16:46:42.042813 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.042885 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 16:46:42.043213 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:42.111441 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 2
I20260928 16:46:42.112623 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.112153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60990)
0928 16:46:42.112366 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.112458 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 16:46:42.113557 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.113253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:60998)
0928 16:46:42.113380 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.113451 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:46:42.113765 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
W20260928 16:46:42.152729 8344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:42.153033 8344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:42.153131 8344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:42.162452 8344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:42.162570 8344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:42.162623 8344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:42.162680 8344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:42.162724 8344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:42.172883 8344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46017,127.26.101.61:41265,127.26.101.60:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.101.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:42.174243 8344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:42.175586 8344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:46:42.183998 8337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 4
I20260928 16:46:42.185184 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.184712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32774)
0928 16:46:42.184932 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.185024 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:46:42.186415 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.185961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32786)
0928 16:46:42.186183 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.186269 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260928 16:46:42.186864 8337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
W20260928 16:46:42.187701 8350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:42.187785 8352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:42.187734 8349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:42.189080 8344 server_base.cc:1034] running on GCE node
I20260928 16:46:42.190057 8344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:42.191691 8344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:42.192896 8344 hybrid_clock.cc:648] HybridClock initialized: now 1790614002192846 us; error 45 us; skew 500 ppm
I20260928 16:46:42.193259 8344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614002, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:42.196480 8344 init.cc:377] Logged in from keytab as kudu/127.26.101.60@KRBTEST.COM (short username kudu)
I20260928 16:46:42.198071 8344 webserver.cc:460] Webserver started at http://127.26.101.60:42667/ using document root <none> and password file <none>
I20260928 16:46:42.198769 8344 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:42.198879 8344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:42.199251 8344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:42.202319 8344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data/instance:
uuid: "6b4118ebd08e462ab1a979266bcba534"
format_stamp: "Formatted at 2026-09-28 16:46:42 on dist-test-slave-t0tw"
server_key: "d9b7e7cfcc7432d9d3b197530120bb06"
server_key_iv: "b761baf14353d9ab328433e28ac7b57f"
server_key_version: "encryptionkey@0"
I20260928 16:46:42.203172 8344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal/instance:
uuid: "6b4118ebd08e462ab1a979266bcba534"
format_stamp: "Formatted at 2026-09-28 16:46:42 on dist-test-slave-t0tw"
server_key: "d9b7e7cfcc7432d9d3b197530120bb06"
server_key_iv: "b761baf14353d9ab328433e28ac7b57f"
server_key_version: "encryptionkey@0"
I20260928 16:46:42.208441 8344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:46:42.211958 8359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:42.213346 8344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:46:42.213546 8344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "6b4118ebd08e462ab1a979266bcba534"
format_stamp: "Formatted at 2026-09-28 16:46:42 on dist-test-slave-t0tw"
server_key: "d9b7e7cfcc7432d9d3b197530120bb06"
server_key_iv: "b761baf14353d9ab328433e28ac7b57f"
server_key_version: "encryptionkey@0"
I20260928 16:46:42.213754 8344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:42.219961 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 3
I20260928 16:46:42.221014 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.220519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32796)
0928 16:46:42.220732 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.220838 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 16:46:42.222084 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.221675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32804)
0928 16:46:42.221876 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.221958 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 16:46:42.222317 8263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:45257: Network error: Client connection negotiation failed: client connection to 127.26.101.60:45257: connect: Connection refused (error 111)
I20260928 16:46:42.237423 8344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:42.241940 8344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:42.242184 8344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:42.260149 8344 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.60:45257
I20260928 16:46:42.260202 8411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.60:45257 every 8 connection(s)
I20260928 16:46:42.261588 8344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
I20260928 16:46:42.265601 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:42.267061 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8344
I20260928 16:46:42.267292 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613789516984-27028-0/minicluster-data/master-2/wal/instance
I20260928 16:46:42.268029 27028 external_mini_cluster.cc:1428] Setting key f39dcde5e65e18f3f99bbd792b0a912c
I20260928 16:46:42.275708 8412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614002, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
I20260928 16:46:42.296355 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.280023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.26.101.62:46017)
0928 16:46:42.280264 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.280268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.281059 (+ 791us) server_negotiation.cc:407] Connection header received
0928 16:46:42.281259 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.281263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.281326 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.281402 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.282996 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.283876 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.285571 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.286066 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.286193 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.289958 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.289965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.289970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.290015 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.292789 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.293659 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.293663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.293665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.293805 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.294265 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.294270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.294273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.294641 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.294780 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.295737 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.295973 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 16:46:42.297676 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.280038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:46017 (local address 127.0.0.1:45650)
0928 16:46:42.280859 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.280919 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:42.281135 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:42.281513 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:42.281522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:42.282389 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:42.282821 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.282834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.284053 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.284059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.285399 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.285417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.286325 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.286333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.286516 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.287155 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:42.287195 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:42.289727 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:42.292935 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.292946 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.292960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.293530 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.293915 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.293920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.293923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.294136 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.294762 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:42.294770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:42.295292 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 16:46:42.295644 (+ 352us) client_negotiation.cc:240] Negotiation successful
0928 16:46:42.296181 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":22,"thread_start_us":379,"threads_started":1}
I20260928 16:46:42.297701 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.270095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.26.101.60:45257)
0928 16:46:42.276117 (+ 6022us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.276131 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.276180 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:46:42.277832 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.277857 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.278315 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.278705 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.280663 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.282836 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.284608 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.285326 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.285466 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.290627 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.290640 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.290660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.290704 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.293872 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.294746 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.294755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.294758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.294927 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.295344 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.295349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.295352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.295769 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.296053 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.296406 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.296637 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":5765,"thread_start_us":5521,"threads_started":1}
I20260928 16:46:42.299422 8412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614002, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:42.313117 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.300182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:41265 (local address 127.0.0.1:56824)
0928 16:46:42.300351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.300369 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:42.300471 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:42.300806 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:42.300810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:42.301209 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:42.301398 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.301410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.302442 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.302448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.303383 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.303395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.304105 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.304111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.304308 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.304934 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:42.304952 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:42.306975 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:42.310631 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.310637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.310642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.311040 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.311449 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.311454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.311458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.311612 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.312139 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:42.312144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:42.312315 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 16:46:42.312435 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 16:46:42.312739 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 16:46:42.313149 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.300262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.26.101.61:41265)
0928 16:46:42.300467 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.300473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.300495 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:46:42.300592 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.300598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.300674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.300756 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.301539 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.302300 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.303520 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.303958 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.304085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.307277 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.307285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.307289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.307334 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.310466 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.311178 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.311183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.311185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.311330 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.311725 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.311729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.311731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.312019 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.312167 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.312563 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.312778 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:46:42.314060 8412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:46:42.316951 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.301209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.26.101.62:46017)
0928 16:46:42.301383 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.301388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.301407 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:46:42.301569 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.301574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.301663 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.301748 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.303153 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.303903 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.305318 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.305733 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.305839 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.311270 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.311276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.311279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.311314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.313977 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.314708 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.314715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.314718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.314896 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.315379 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.315386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.315389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.315787 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.316011 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.316380 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.316632 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614002, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:46:42.329429 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.314909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32814)
0928 16:46:42.315135 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.315154 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:42.315251 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:42.315651 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:42.315655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:42.316156 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:42.316340 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.316352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.317970 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.317975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.319149 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.319164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.319963 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.319971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.320173 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.320765 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:42.320788 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:42.323210 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:42.326865 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.326870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.326873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.327324 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.327699 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.327703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.327707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.327857 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.328391 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:42.328396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:42.328544 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 16:46:42.328688 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 16:46:42.329028 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 16:46:42.329478 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.315204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.26.101.60:45257)
0928 16:46:42.315349 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.315354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.315369 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:46:42.315414 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.315424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.315495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.315592 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.316485 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.317824 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.319281 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.319788 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.319898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.323412 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.323417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.323420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.323451 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.326703 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.327452 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.327456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.327458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.327589 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.327960 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.327964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.327966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.328281 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.328413 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.328805 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.329217 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":196,"server-negotiator.queue_time_us":44}
I20260928 16:46:42.334566 8412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: Bootstrap starting.
I20260928 16:46:42.335191 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.318755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.26.101.61:41265)
0928 16:46:42.318918 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.318923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.318938 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:46:42.319107 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.319112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.319185 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.319282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.320986 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.321803 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.323288 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.323739 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.323838 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.328740 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.328747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.328751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.328795 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.332445 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.333150 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.333156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.333160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.333360 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.333748 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.333752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.333754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.334029 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.334316 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.334600 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.334837 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:46:42.338388 8412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:42.339624 8412 log.cc:824] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:42.340242 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:46:42.342473 8412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: No bootstrap required, opened a new log
I20260928 16:46:42.353062 8412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.353628 8412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:42.353802 8412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4118ebd08e462ab1a979266bcba534, State: Initialized, Role: FOLLOWER
I20260928 16:46:42.354696 8412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.362040 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.365180 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.364492 8412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:42.367823 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.345908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.26.101.62:46017)
0928 16:46:42.347028 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.347035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.347668 (+ 633us) server_negotiation.cc:407] Connection header received
0928 16:46:42.347859 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.347865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.347960 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.348071 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.350656 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.351923 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.354977 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.355698 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.355826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.360063 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.360070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.360075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.360135 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.363983 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.365142 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.365148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.365151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.365327 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.365852 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.365857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.365860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.366238 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.366495 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.367198 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.367455 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984}
I20260928 16:46:42.367833 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.346114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.26.101.61:41265)
0928 16:46:42.346295 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.346302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.348369 (+ 2067us) server_negotiation.cc:407] Connection header received
0928 16:46:42.348880 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.348887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.348970 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.349037 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.351496 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.352590 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.355021 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.355714 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.355808 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.360063 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.360070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.360075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.360124 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.363663 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.365138 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.365144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.365147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.365328 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.365859 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.365864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.365867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.366262 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.366498 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.367198 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.367454 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:46:42.373157 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.345402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.26.101.60:45257)
0928 16:46:42.345593 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.345603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.354854 (+ 9251us) server_negotiation.cc:407] Connection header received
0928 16:46:42.355109 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.355114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.355196 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.355268 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.357867 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.359140 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.361407 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.362115 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.362223 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.365717 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.365724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.365728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.365768 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.369587 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.370903 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.370907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.370910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.371069 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.371488 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.371494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.371501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.371866 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.372127 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.372438 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.372689 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 16:46:42.380311 8436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:46:42.380523 8436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:46:42.377820 8344 master_runner.cc:186] Error getting master registration for 127.26.101.62:46017: OK, instance_id { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" instance_seqno: 1790614001259792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:46:42.389276 8344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:46:42.399531 8263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 4
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
I20260928 16:46:42.414361 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.400496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.26.101.60:45257)
0928 16:46:42.400695 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.400702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.400723 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:46:42.400785 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.400789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.400877 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.400965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.401785 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.402814 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.404486 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.404935 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.405050 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.408452 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.408458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.408461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.408496 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.411358 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.412208 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.412213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.412216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.412432 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.412887 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.412892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.412895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.413275 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.413428 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.413900 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.414104 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:46:42.414597 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.400273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32842)
0928 16:46:42.400479 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.400497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:42.400634 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:42.401024 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:42.401029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:42.401448 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:42.401636 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.401645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.402978 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.402983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.404347 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.404362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.405084 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.405091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.405277 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.405850 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:42.405868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:42.408202 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:42.411507 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.411513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.411517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.412044 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.412584 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.412588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.412591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.412776 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.413421 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:42.413426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:42.413636 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 16:46:42.413800 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 16:46:42.414118 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:46:42.419504 8263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: Bootstrap starting.
I20260928 16:46:42.422771 8263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:42.425853 8263 log.cc:824] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:42.429638 8263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: No bootstrap required, opened a new log
I20260928 16:46:42.440846 8263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.441313 8263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:42.441448 8263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee8c6848af443399ad499c7c6d50ce9, State: Initialized, Role: FOLLOWER
I20260928 16:46:42.442180 8263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.443318 8439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.443661 8439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.444617 8263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:46:42.455482 8450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:46:42.455776 8450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:46:42.455482 8195 master_runner.cc:186] Error getting master registration for 127.26.101.61:41265: OK, instance_id { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" instance_seqno: 1790614001757299 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:46:42.463856 8195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:46:42.486160 8337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } attempt: 5
Sep 28 16:46:42 dist-test-slave-t0tw krb5kdc[8182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614001, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
W20260928 16:46:42.494438 8365 tablet.cc:2369] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:46:42.499754 8217 tablet.cc:2369] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:42.502188 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.486863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:45257 (local address 127.0.0.1:32854)
0928 16:46:42.487182 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:42.487203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:42.487345 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:42.487690 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:42.487695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:42.488143 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:42.488310 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.488320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.489365 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.489371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.490282 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:42.490295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.491033 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.491047 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:42.491301 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.491881 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:42.491898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:42.495882 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:42.499367 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.499372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.499374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.499940 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.500436 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:42.500441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:42.500445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:42.500620 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:42.501188 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:42.501193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:42.501347 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 16:46:42.501480 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:46:42.501806 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 16:46:42.502383 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:42.486949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.26.101.60:45257)
0928 16:46:42.487190 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:42.487196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:42.487281 (+ 85us) server_negotiation.cc:407] Connection header received
0928 16:46:42.487470 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:42.487475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:42.487553 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:42.487637 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:42.488477 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.489211 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.490424 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:42.490883 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:42.490997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:42.496118 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:42.496125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:42.496129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:42.496170 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:42.499223 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.500112 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.500118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.500121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.500274 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:42.500752 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:42.500755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:42.500758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:42.501065 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:42.501221 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:42.501600 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 16:46:42.501842 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 16:46:42.506930 8337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: Bootstrap starting.
I20260928 16:46:42.510242 8337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:42.513252 8337 log.cc:824] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:42.516078 8337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: No bootstrap required, opened a new log
I20260928 16:46:42.523815 8337 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.524191 8337 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:42.524315 8337 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0de179e2c784be38f65fa2f8ff11cf1, State: Initialized, Role: FOLLOWER
I20260928 16:46:42.524957 8337 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.525949 8453 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.526468 8453 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.527292 8337 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:42.555055 8270 master_runner.cc:263] No leader master found from master d0de179e2c784be38f65fa2f8ff11cf1
I20260928 16:46:42.556463 8270 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:46:42.559320 8465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:46:42.559574 8465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:46:42.560315 8453 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:46:42.560614 8453 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.562342 8453 leader_election.cc:288] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee8c6848af443399ad499c7c6d50ce9 (127.26.101.62:46017), 6b4118ebd08e462ab1a979266bcba534 (127.26.101.60:45257)
I20260928 16:46:42.564119 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee8c6848af443399ad499c7c6d50ce9" is_pre_election: true
I20260928 16:46:42.564839 8238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0de179e2c784be38f65fa2f8ff11cf1 in term 0.
I20260928 16:46:42.565737 8387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4118ebd08e462ab1a979266bcba534" is_pre_election: true
I20260928 16:46:42.566428 8387 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0de179e2c784be38f65fa2f8ff11cf1 in term 0.
I20260928 16:46:42.567380 8287 leader_election.cc:302] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4118ebd08e462ab1a979266bcba534, d0de179e2c784be38f65fa2f8ff11cf1; no voters:
I20260928 16:46:42.568631 8453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:46:42.568789 8453 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:46:42.568898 8453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:42.572779 8453 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.579300 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee8c6848af443399ad499c7c6d50ce9"
I20260928 16:46:42.579581 8238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:42.583529 8238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0de179e2c784be38f65fa2f8ff11cf1 in term 1.
I20260928 16:46:42.584208 8288 leader_election.cc:302] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee8c6848af443399ad499c7c6d50ce9, d0de179e2c784be38f65fa2f8ff11cf1; no voters:
I20260928 16:46:42.588070 8387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4118ebd08e462ab1a979266bcba534"
I20260928 16:46:42.588351 8387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:42.592173 8387 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0de179e2c784be38f65fa2f8ff11cf1 in term 1.
I20260928 16:46:42.593183 8453 leader_election.cc:288] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee8c6848af443399ad499c7c6d50ce9 (127.26.101.62:46017), 6b4118ebd08e462ab1a979266bcba534 (127.26.101.60:45257)
I20260928 16:46:42.593653 8453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:42.594135 8453 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [term 1 LEADER]: Becoming Leader. State: Replica: d0de179e2c784be38f65fa2f8ff11cf1, State: Running, Role: LEADER
I20260928 16:46:42.595114 8453 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } }
I20260928 16:46:42.613458 8467 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0de179e2c784be38f65fa2f8ff11cf1. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.614044 8467 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:42.621500 8469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:42.624049 8469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:42.652690 8387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [term 1 FOLLOWER]: Refusing update from remote peer d0de179e2c784be38f65fa2f8ff11cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:42.652702 8238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [term 1 FOLLOWER]: Refusing update from remote peer d0de179e2c784be38f65fa2f8ff11cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:42.653779 8453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:42.661170 8453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:42.695499 8439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0de179e2c784be38f65fa2f8ff11cf1. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.695885 8439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.699925 8470 mvcc.cc:205] Tried to move back new op lower bound from 7334354954853658624 to 7334354954657280000. Current Snapshot: MvccSnapshot[applied={T|T < 7334354954853658624}]
I20260928 16:46:42.711355 8471 mvcc.cc:205] Tried to move back new op lower bound from 7334354954853658624 to 7334354954657280000. Current Snapshot: MvccSnapshot[applied={T|T < 7334354954853658624}]
I20260928 16:46:42.712872 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0de179e2c784be38f65fa2f8ff11cf1. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.713271 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.716657 8467 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.716939 8467 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:42.724135 8453 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.725122 8453 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:42.726109 8472 mvcc.cc:205] Tried to move back new op lower bound from 7334354954853658624 to 7334354954657280000. Current Snapshot: MvccSnapshot[applied={T|T < 7334354954853658624}]
I20260928 16:46:42.730034 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.730329 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.733246 8439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ee8c6848af443399ad499c7c6d50ce9" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46017 } } peers { permanent_uuid: "d0de179e2c784be38f65fa2f8ff11cf1" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 41265 } } peers { permanent_uuid: "6b4118ebd08e462ab1a979266bcba534" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 45257 } } }
I20260928 16:46:42.733492 8439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:46:42.737866 8469 catalog_manager.cc:1347] Generated new cluster ID: 0a2a7b440ddc40ac8edbbc5ace05026c
I20260928 16:46:42.737987 8469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:42.771935 8469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:42.773902 8469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:42.810657 8469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0de179e2c784be38f65fa2f8ff11cf1: Generated new TSK 0
I20260928 16:46:42.811908 8469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:46:43.382604 8436 catalog_manager.cc:1259] Loaded cluster ID: 0a2a7b440ddc40ac8edbbc5ace05026c
I20260928 16:46:43.382725 8436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: loading cluster ID for follower catalog manager: success
I20260928 16:46:43.385305 8436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: acquiring CA information for follower catalog manager: success
I20260928 16:46:43.387209 8436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b4118ebd08e462ab1a979266bcba534: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:46:43.457883 8450 catalog_manager.cc:1259] Loaded cluster ID: 0a2a7b440ddc40ac8edbbc5ace05026c
I20260928 16:46:43.457995 8450 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: loading cluster ID for follower catalog manager: success
I20260928 16:46:43.460575 8450 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: acquiring CA information for follower catalog manager: success
I20260928 16:46:43.462446 8450 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ee8c6848af443399ad499c7c6d50ce9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:46:43.948480 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8195
I20260928 16:46:43.949708 8258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:44.063566 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8195
I20260928 16:46:44.083379 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8270
I20260928 16:46:44.084529 8332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:44.198879 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8270
I20260928 16:46:44.217392 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8344
I20260928 16:46:44.218613 8407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:44.327179 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8344
2026-09-28T16:46:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4783 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:46:44 dist-test-slave-t0tw krb5kdc[8490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:46:44 dist-test-slave-t0tw krb5kdc[8490](info): set up 2 sockets
Sep 28 16:46:44 dist-test-slave-t0tw krb5kdc[8490](info): commencing operation
krb5kdc: starting...
W20260928 16:46:45.351415 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:46:45 dist-test-slave-t0tw krb5kdc[8490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:46:45.598742 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:36239
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:45.948216 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:45.948583 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:45.948715 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:45.956238 8503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:45.956393 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:45.956465 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:45.956542 8503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:45.956604 8503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:45.967155 8503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:45.969282 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:45.971378 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:45.984449 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:45.984431 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:45.984496 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:45.985508 8503 server_base.cc:1034] running on GCE node
I20260928 16:46:45.986611 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:45.988373 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:45.989638 8503 hybrid_clock.cc:648] HybridClock initialized: now 1790614005989574 us; error 47 us; skew 500 ppm
I20260928 16:46:45.990044 8503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:45 dist-test-slave-t0tw krb5kdc[8490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614005, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:45.993464 8503 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:45.995366 8503 webserver.cc:460] Webserver started at http://127.26.101.62:41737/ using document root <none> and password file <none>
I20260928 16:46:45.996161 8503 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:45.996295 8503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:45.996621 8503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:45.999989 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "c7492c5913404d87a8446fca31870b3d"
format_stamp: "Formatted at 2026-09-28 16:46:45 on dist-test-slave-t0tw"
server_key: "3de6c2f231f5ddc1e69413ce9b0c78e5"
server_key_iv: "1ed3d6881f7f43abeec192185655ea58"
server_key_version: "encryptionkey@0"
I20260928 16:46:46.000836 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "c7492c5913404d87a8446fca31870b3d"
format_stamp: "Formatted at 2026-09-28 16:46:45 on dist-test-slave-t0tw"
server_key: "3de6c2f231f5ddc1e69413ce9b0c78e5"
server_key_iv: "1ed3d6881f7f43abeec192185655ea58"
server_key_version: "encryptionkey@0"
I20260928 16:46:46.006726 8503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:46:46.010812 8518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:46.012480 8503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:46.012689 8503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "c7492c5913404d87a8446fca31870b3d"
format_stamp: "Formatted at 2026-09-28 16:46:45 on dist-test-slave-t0tw"
server_key: "3de6c2f231f5ddc1e69413ce9b0c78e5"
server_key_iv: "1ed3d6881f7f43abeec192185655ea58"
server_key_version: "encryptionkey@0"
I20260928 16:46:46.012899 8503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:46.044983 8503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:46.050032 8503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:46.050355 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:46.069511 8503 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:36239
I20260928 16:46:46.069514 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:36239 every 8 connection(s)
I20260928 16:46:46.071017 8503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:46.074390 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8503
I20260928 16:46:46.074610 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:46:46.075314 8571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:46.075377 27028 external_mini_cluster.cc:1428] Setting key 17cce8d81bdff7ebccbe39e4b12652cf
I20260928 16:46:46.090549 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d: Bootstrap starting.
Sep 28 16:46:46 dist-test-slave-t0tw krb5kdc[8490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:46.094323 8571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:46.095440 8571 log.cc:824] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:46.098893 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d: No bootstrap required, opened a new log
I20260928 16:46:46.100833 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:46.077214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.26.101.62:36239)
0928 16:46:46.078053 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:46.078065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:46.078114 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:46:46.080223 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:46.080251 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:46.080723 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:46.081109 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:46.083224 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:46.084665 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:46.086645 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:46.087571 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:46.087751 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:46.093007 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:46.093016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:46.093034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:46.093082 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:46.096454 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:46.097377 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:46.097384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:46.097391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:46.097564 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:46.098008 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:46.098012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:46.098014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:46.098584 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:46.098810 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:46.099357 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 16:46:46.099742 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":362,"threads_started":1}
I20260928 16:46:46.105002 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:46:46.108291 8571 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } }
I20260928 16:46:46.108704 8571 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:46.108819 8571 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7492c5913404d87a8446fca31870b3d, State: Initialized, Role: FOLLOWER
I20260928 16:46:46.109450 8571 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } }
I20260928 16:46:46.109699 8571 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:46.109829 8571 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:46.110010 8571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:46.112368 8571 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } }
I20260928 16:46:46.112880 8571 leader_election.cc:302] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7492c5913404d87a8446fca31870b3d; no voters:
I20260928 16:46:46.113472 8571 leader_election.cc:288] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:46:46.113620 8576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:46.114287 8576 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [term 1 LEADER]: Becoming Leader. State: Replica: c7492c5913404d87a8446fca31870b3d, State: Running, Role: LEADER
I20260928 16:46:46.114817 8576 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } }
I20260928 16:46:46.115589 8571 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:46.117654 8577 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7492c5913404d87a8446fca31870b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } } }
I20260928 16:46:46.117687 8578 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7492c5913404d87a8446fca31870b3d. Latest consensus state: current_term: 1 leader_uuid: "c7492c5913404d87a8446fca31870b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7492c5913404d87a8446fca31870b3d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 36239 } } }
I20260928 16:46:46.118120 8577 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:46.118120 8578 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:46.119714 8582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:46.125578 8582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:46.134950 8582 catalog_manager.cc:1347] Generated new cluster ID: 9914b4e9e84442a394b44e4a99105340
I20260928 16:46:46.135118 8582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:46.160451 8582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:46.162508 8582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:46.177259 8582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7492c5913404d87a8446fca31870b3d: Generated new TSK 0
I20260928 16:46:46.178437 8582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:46:46.219889 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8503
I20260928 16:46:46.221328 8566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:46.319991 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8503
2026-09-28T16:46:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1988 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:46:46 dist-test-slave-t0tw krb5kdc[8599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:46:46 dist-test-slave-t0tw krb5kdc[8599](info): set up 2 sockets
Sep 28 16:46:46 dist-test-slave-t0tw krb5kdc[8599](info): commencing operation
krb5kdc: starting...
W20260928 16:46:47.343216 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:46:47 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:46:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:46:47.591750 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:47.930622 8612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:47.930958 8612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:47.931108 8612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:47.939150 8612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:47.939303 8612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:47.939384 8612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:47.939461 8612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:47.939530 8612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:47.950099 8612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:47.951620 8612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:47.953001 8612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:47.966199 8618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:47.966181 8617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:47.966281 8620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:47.967054 8612 server_base.cc:1034] running on GCE node
I20260928 16:46:47.968010 8612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:47.969652 8612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:47.970916 8612 hybrid_clock.cc:648] HybridClock initialized: now 1790614007970847 us; error 58 us; skew 500 ppm
I20260928 16:46:47.971349 8612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:47 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:47.974691 8612 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:47.976447 8612 webserver.cc:460] Webserver started at http://127.26.101.62:39509/ using document root <none> and password file <none>
I20260928 16:46:47.977201 8612 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:47.977335 8612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:47.977648 8612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:47.981065 8612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "3e1b5cf2cc164c1ea332a80b4398d181"
format_stamp: "Formatted at 2026-09-28 16:46:47 on dist-test-slave-t0tw"
server_key: "2206856fe301298583f4f132b5b3f5aa"
server_key_iv: "1bdfc11d256814949ac1b737440069bc"
server_key_version: "encryptionkey@0"
I20260928 16:46:47.981930 8612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "3e1b5cf2cc164c1ea332a80b4398d181"
format_stamp: "Formatted at 2026-09-28 16:46:47 on dist-test-slave-t0tw"
server_key: "2206856fe301298583f4f132b5b3f5aa"
server_key_iv: "1bdfc11d256814949ac1b737440069bc"
server_key_version: "encryptionkey@0"
I20260928 16:46:47.987779 8612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:46:47.991847 8627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:47.993559 8612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:46:47.993783 8612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "3e1b5cf2cc164c1ea332a80b4398d181"
format_stamp: "Formatted at 2026-09-28 16:46:47 on dist-test-slave-t0tw"
server_key: "2206856fe301298583f4f132b5b3f5aa"
server_key_iv: "1bdfc11d256814949ac1b737440069bc"
server_key_version: "encryptionkey@0"
I20260928 16:46:47.994029 8612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:48.014209 8612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:48.018850 8612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:48.019210 8612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:48.037753 8612 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45955
I20260928 16:46:48.037792 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45955 every 8 connection(s)
I20260928 16:46:48.039357 8612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:48.043537 8682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:48.046588 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8612
I20260928 16:46:48.046826 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:46:48.047564 27028 external_mini_cluster.cc:1428] Setting key 082caf45c92b03afa9dedb189f99df80
I20260928 16:46:48.056317 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap starting.
I20260928 16:46:48.059757 8682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:48.060891 8682 log.cc:824] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:46:48 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:48.064713 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: No bootstrap required, opened a new log
I20260928 16:46:48.072290 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:48.049801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.26.101.62:45955)
0928 16:46:48.050675 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:48.050687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:48.050733 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:46:48.052614 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:48.052641 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:48.053113 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:48.053528 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:48.055144 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.056536 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.057900 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.058640 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.058822 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:48.063890 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:48.063901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:48.063919 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:48.063967 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:48.067426 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:48.068457 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:48.068465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:48.068473 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:48.068688 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:48.069160 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:48.069167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:48.069169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:48.069824 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:48.070089 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:48.070548 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 16:46:48.070971 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":394,"threads_started":1}
I20260928 16:46:48.074311 8682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:48.074705 8682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:48.074822 8682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Initialized, Role: FOLLOWER
I20260928 16:46:48.075438 8682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:48.075740 8682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:48.075882 8682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:48.076030 8682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:48.078222 8682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:48.078734 8682 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1b5cf2cc164c1ea332a80b4398d181; no voters:
I20260928 16:46:48.079429 8682 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:46:48.079551 8687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:48.080219 8687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 LEADER]: Becoming Leader. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Running, Role: LEADER
I20260928 16:46:48.080751 8687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:48.081489 8682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:48.083590 8689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1b5cf2cc164c1ea332a80b4398d181. Latest consensus state: current_term: 1 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:48.083954 8689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:48.084649 8688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:48.084947 8688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:48.085536 8694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:48.090679 8694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:48.100608 8694 catalog_manager.cc:1347] Generated new cluster ID: 9110b440f3bd49a28a5e51665b97871c
I20260928 16:46:48.100749 8694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:48.121503 8694 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:46:48.123561 8694 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:48.138509 8694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Generated new TSK 0
I20260928 16:46:48.139789 8694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:46:48.214840 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:48.558540 8708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:48.558866 8708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:48.559011 8708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:48.566802 8708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:48.566948 8708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:48.567145 8708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:46:48.577481 8708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:48.578878 8708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:48.580374 8708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:48.594588 8713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:48.594636 8714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:48.594666 8716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:48.595497 8708 server_base.cc:1034] running on GCE node
I20260928 16:46:48.596133 8708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:48.597082 8708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:48.598296 8708 hybrid_clock.cc:648] HybridClock initialized: now 1790614008598251 us; error 36 us; skew 500 ppm
I20260928 16:46:48.598606 8708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:48 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614008, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:48.601958 8708 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:46:48.603528 8708 webserver.cc:460] Webserver started at http://127.26.101.1:33001/ using document root <none> and password file <none>
I20260928 16:46:48.604251 8708 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:48.604362 8708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:48.604657 8708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:48.607973 8708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "ecc027b7b675497cb9f774ff38377a74"
format_stamp: "Formatted at 2026-09-28 16:46:48 on dist-test-slave-t0tw"
server_key: "311cad86112bcd78342482ff7bb322f4"
server_key_iv: "a89217e3a3f34cb0d1119fa162a78fad"
server_key_version: "encryptionkey@0"
I20260928 16:46:48.608801 8708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "ecc027b7b675497cb9f774ff38377a74"
format_stamp: "Formatted at 2026-09-28 16:46:48 on dist-test-slave-t0tw"
server_key: "311cad86112bcd78342482ff7bb322f4"
server_key_iv: "a89217e3a3f34cb0d1119fa162a78fad"
server_key_version: "encryptionkey@0"
I20260928 16:46:48.614464 8708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:46:48.618304 8723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:48.619931 8708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:48.620119 8708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "ecc027b7b675497cb9f774ff38377a74"
format_stamp: "Formatted at 2026-09-28 16:46:48 on dist-test-slave-t0tw"
server_key: "311cad86112bcd78342482ff7bb322f4"
server_key_iv: "a89217e3a3f34cb0d1119fa162a78fad"
server_key_version: "encryptionkey@0"
I20260928 16:46:48.620319 8708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:48.645730 8708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:48.650271 8708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:48.650563 8708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:48.651705 8708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:48.653132 8708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:48.653210 8708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:48.653311 8708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:48.653365 8708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:48.685557 8708 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:46871
I20260928 16:46:48.685619 8837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:46871 every 8 connection(s)
I20260928 16:46:48.687129 8708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:48.692378 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8708
I20260928 16:46:48.692595 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:46:48.693234 27028 external_mini_cluster.cc:1428] Setting key 1b3687ac3b01e7521e0ea8d5519908de
Sep 28 16:46:48 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614008, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:48.708359 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:48.689960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:49939 (local address 127.26.101.62:45955)
0928 16:46:48.690480 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:48.690484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:48.691572 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 16:46:48.693530 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:48.693535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:48.693632 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:48.693735 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:48.695685 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.696430 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.698048 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.698514 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.698634 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:48.702470 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:48.702476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:48.702479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:48.702515 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:48.705044 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:48.705868 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:48.705872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:48.705875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:48.706003 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:48.706456 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:48.706460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:48.706462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:48.706746 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:48.706888 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:48.707718 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 16:46:48.707976 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260928 16:46:48.709376 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:48.690583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:49939)
0928 16:46:48.691400 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:48.691441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:48.693204 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:48.694004 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:48.694014 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:48.694768 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:48.695501 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:48.695533 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.696593 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.696597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:48.697877 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:48.697890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:48.698753 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:48.698761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:48.699010 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:48.699793 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:48.699818 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:48.702217 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:48.705214 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:48.705225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:48.705245 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:48.705727 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:48.706117 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:48.706123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:48.706127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:48.706335 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:48.706885 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:48.706894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:48.707320 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 16:46:48.707580 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 16:46:48.708170 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":282,"threads_started":1}
I20260928 16:46:48.711180 8838 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:48.711519 8838 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:48.712285 8838 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:48.714867 8644 ts_manager.cc:194] Registered new tserver with Master: ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871)
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:46:48.716691 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:49939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:46:48.747761 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:49.080375 8844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:49.080689 8844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:49.080786 8844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:49.088310 8844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:49.088439 8844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:49.088557 8844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:46:49.098706 8844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:49.100309 8844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:49.101635 8844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:49.114802 8849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:49.114861 8850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:49.115200 8852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:49.116667 8844 server_base.cc:1034] running on GCE node
I20260928 16:46:49.117333 8844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:49.118373 8844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:49.119606 8844 hybrid_clock.cc:648] HybridClock initialized: now 1790614009119543 us; error 57 us; skew 500 ppm
I20260928 16:46:49.119941 8844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:49 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:49.123354 8844 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:46:49.124951 8844 webserver.cc:460] Webserver started at http://127.26.101.2:39423/ using document root <none> and password file <none>
I20260928 16:46:49.125703 8844 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:49.125811 8844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:49.126065 8844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:49.129063 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "4a760c3939744e3f906f1b6245a65672"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "e7fcb73c8607b90a152771cef31ee45d"
server_key_iv: "8c35bbcde8b3f0b8b5466c00220cdaf5"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.129834 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "4a760c3939744e3f906f1b6245a65672"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "e7fcb73c8607b90a152771cef31ee45d"
server_key_iv: "8c35bbcde8b3f0b8b5466c00220cdaf5"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.135545 8844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:46:49.139392 8859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.141036 8844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:46:49.141209 8844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "4a760c3939744e3f906f1b6245a65672"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "e7fcb73c8607b90a152771cef31ee45d"
server_key_iv: "8c35bbcde8b3f0b8b5466c00220cdaf5"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.141413 8844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:49.161410 8844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:49.166126 8844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:49.166411 8844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:49.167444 8844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:49.168852 8844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:49.168929 8844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.169054 8844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:49.169103 8844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.201361 8844 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:34101
I20260928 16:46:49.201403 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:34101 every 8 connection(s)
I20260928 16:46:49.202651 8844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:49.203940 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8844
I20260928 16:46:49.204289 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:46:49.204989 27028 external_mini_cluster.cc:1428] Setting key cdd69d16ac2d93203f0d5be4d934ce77
Sep 28 16:46:49 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:49.225334 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.205897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:49447 (local address 127.26.101.62:45955)
0928 16:46:49.206128 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.206133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.207557 (+ 1424us) server_negotiation.cc:407] Connection header received
0928 16:46:49.209577 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.209583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.209670 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.209773 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:49.211616 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.212565 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.214212 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.214662 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.214762 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.219017 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:49.219024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:49.219029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:49.219106 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:49.221679 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.222604 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.222608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.222611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.222750 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.223263 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.223267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.223269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.223582 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:49.223739 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.224631 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.224930 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:46:49.226262 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.206458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:49447)
0928 16:46:49.207356 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:49.207406 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:49.209282 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:49.209981 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:49.209991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:49.210753 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:49.211435 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.211455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.212737 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.212741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.214055 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.214069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.214914 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.214920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.215259 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.216032 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:49.216065 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:49.218724 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:49.221829 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:49.221839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:49.221858 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:49.222441 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:49.222853 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:49.222858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:49.222862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:49.223154 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:49.223720 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:49.223729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:49.224154 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 16:46:49.224473 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 16:46:49.224982 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":327,"threads_started":1}
I20260928 16:46:49.227804 8973 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:49.228101 8973 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
I20260928 16:46:49.228888 8973 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:49.230444 8644 ts_manager.cc:194] Registered new tserver with Master: 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
I20260928 16:46:49.231575 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:49447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:46:49.259626 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:49.596511 8978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:49.596825 8978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:49.596952 8978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:49.604923 8978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:49.605049 8978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:49.605206 8978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:46:49.615449 8978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:49.616876 8978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:49.618371 8978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:49.634423 8984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:49.634423 8983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:49.634712 8986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:49.635586 8978 server_base.cc:1034] running on GCE node
I20260928 16:46:49.636422 8978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:49.637655 8978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:49.638904 8978 hybrid_clock.cc:648] HybridClock initialized: now 1790614009638837 us; error 57 us; skew 500 ppm
I20260928 16:46:49.639386 8978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:49 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:49.644029 8978 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:46:49.646023 8978 webserver.cc:460] Webserver started at http://127.26.101.3:39301/ using document root <none> and password file <none>
I20260928 16:46:49.646828 8978 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:49.646963 8978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:49.647353 8978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:46:49.650543 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "b6bf48241ac24dabb59f53e88744bb09"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "93749f0f595a7384b88425a9589d45f4"
server_key_iv: "f53ee6f05c6c314d625327aabe9cbfcd"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.651449 8978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "b6bf48241ac24dabb59f53e88744bb09"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "93749f0f595a7384b88425a9589d45f4"
server_key_iv: "f53ee6f05c6c314d625327aabe9cbfcd"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.657364 8978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 16:46:49.661413 8993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.663209 8978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:46:49.663405 8978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "b6bf48241ac24dabb59f53e88744bb09"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "93749f0f595a7384b88425a9589d45f4"
server_key_iv: "f53ee6f05c6c314d625327aabe9cbfcd"
server_key_version: "encryptionkey@0"
I20260928 16:46:49.663632 8978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:49.675143 8978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:49.679920 8978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:49.680217 8978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:49.681315 8978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:49.682845 8978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:49.682926 8978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.683022 8978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:49.683106 8978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:49.716342 8978 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:39827
I20260928 16:46:49.716365 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:39827 every 8 connection(s)
I20260928 16:46:49.718086 8978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:49.720153 8838 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:49.723637 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 8978
I20260928 16:46:49.723819 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:46:49.724344 27028 external_mini_cluster.cc:1428] Setting key b95eb525737059ae92ae0f8372b76fde
Sep 28 16:46:49 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614009, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:49.742765 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.721399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:45259 (local address 127.26.101.62:45955)
0928 16:46:49.721611 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.721617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.723196 (+ 1579us) server_negotiation.cc:407] Connection header received
0928 16:46:49.725918 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.725924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.726017 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.726120 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:49.728392 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.729167 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.730795 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.731261 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.731380 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.735686 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:49.735692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:49.735695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:49.735735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:49.738838 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.739931 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.739936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.739940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.740131 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.740697 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.740703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.740707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.741100 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:49.741265 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.742113 (+ 848us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.742377 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:46:49.743664 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.722014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:45259)
0928 16:46:49.722979 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:49.723034 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:49.725582 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:49.726421 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:49.726435 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:49.727403 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:49.728189 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.728213 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.729333 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.729338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.730628 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.730644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.731506 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.731514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.731770 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.732729 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:49.732760 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:49.735419 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:49.739042 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:49.739051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:49.739066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:49.739779 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:49.740293 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:49.740298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:49.740302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:49.740574 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:49.741253 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:49.741260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:49.741688 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 16:46:49.741955 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 16:46:49.742457 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":348,"threads_started":1}
I20260928 16:46:49.745220 9106 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:49.745563 9106 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:49.746373 9106 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:49.747992 8644 ts_manager.cc:194] Registered new tserver with Master: b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827)
I20260928 16:46:49.748817 8644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:45259
I20260928 16:46:49.754515 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:46:49.776363 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.761222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.26.101.62:45955)
0928 16:46:49.762489 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.762494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.762515 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:46:49.762575 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.762579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.762653 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.762742 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:49.764720 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.765468 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.767518 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.767981 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.768085 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.770746 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:49.770752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:49.770755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:49.770794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:49.773557 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.774318 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.774323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.774325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.774473 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:49.774845 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:49.774848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:49.774851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:49.775235 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:49.775488 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.775819 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.776041 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099}
I20260928 16:46:49.781708 8644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:46:49.788975 8644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:46:49.825127 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.802893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:46871 (local address 127.0.0.1:53436)
0928 16:46:49.812435 (+ 9542us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:49.812478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:49.812681 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:49.814152 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:49.814162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:49.814204 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:49.814671 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.814686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.818432 (+ 3746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.818438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.821198 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.821239 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.823741 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.823750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.824023 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.824110 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 16:46:49.824266 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 16:46:49.824509 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9260,"mutex_wait_us":1,"thread_start_us":16585,"threads_started":1}
I20260928 16:46:49.827678 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.803159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.26.101.1:46871)
0928 16:46:49.813378 (+ 10219us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.813387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.813419 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:46:49.813530 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.813539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.813851 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.814098 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:49.814938 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.818265 (+ 3327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.821444 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.823359 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.823505 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.823772 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.825671 (+ 1899us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.825907 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10030,"thread_start_us":9869,"threads_started":1}
I20260928 16:46:49.828902 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.814518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34101 (local address 127.0.0.1:34588)
0928 16:46:49.819655 (+ 5137us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:49.819680 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:49.819902 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:49.820757 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:49.820765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:49.820792 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:49.821212 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.821221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.823514 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.823520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.825735 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.825750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.828023 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.828030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.828228 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.828263 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:46:49.828354 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:46:49.828504 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4942,"mutex_wait_us":2,"thread_start_us":4255,"threads_started":1}
I20260928 16:46:49.829149 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.813283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.26.101.2:34101)
0928 16:46:49.817764 (+ 4481us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.817773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.819789 (+ 2016us) server_negotiation.cc:407] Connection header received
0928 16:46:49.820128 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.820134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.820360 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.820573 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:49.821600 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.823331 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.825942 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.827456 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.827610 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.827833 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.828597 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.828794 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4332,"thread_start_us":4225,"threads_started":1}
I20260928 16:46:49.830677 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.803208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:39827 (local address 127.0.0.1:52806)
0928 16:46:49.818530 (+ 15322us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:49.818554 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:49.818697 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:49.820281 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:49.820286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:49.820310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:49.820794 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.820810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.822814 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.822819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.825271 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:49.825289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.829810 (+ 4521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.829817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:49.829993 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.830029 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:46:49.830140 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:46:49.830289 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15187,"mutex_wait_us":98,"thread_start_us":8794,"threads_started":1}
I20260928 16:46:49.831617 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:49.803581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.26.101.3:39827)
0928 16:46:49.819589 (+ 16008us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:49.819599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:49.819638 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:46:49.819744 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:49.819753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:49.820074 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:49.820285 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:49.820998 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.822649 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.825487 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:49.829568 (+ 4081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:49.829721 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:49.829969 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:49.831106 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0928 16:46:49.831299 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15824,"thread_start_us":15668,"threads_started":1}
I20260928 16:46:49.833010 8772 tablet_service.cc:1467] Processing CreateTablet for tablet cce226ef09b14e7da3a85cd857e57e11 (DEFAULT_TABLE table=test-table [id=5cd8f43779d7402da4434d4b39917a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:49.834347 8772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce226ef09b14e7da3a85cd857e57e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:49.835234 8907 tablet_service.cc:1467] Processing CreateTablet for tablet cce226ef09b14e7da3a85cd857e57e11 (DEFAULT_TABLE table=test-table [id=5cd8f43779d7402da4434d4b39917a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:49.835234 9041 tablet_service.cc:1467] Processing CreateTablet for tablet cce226ef09b14e7da3a85cd857e57e11 (DEFAULT_TABLE table=test-table [id=5cd8f43779d7402da4434d4b39917a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:49.836833 8907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce226ef09b14e7da3a85cd857e57e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:49.836829 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce226ef09b14e7da3a85cd857e57e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:49.858829 9126 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Bootstrap starting.
I20260928 16:46:49.862414 9125 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Bootstrap starting.
I20260928 16:46:49.863467 9127 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Bootstrap starting.
I20260928 16:46:49.867602 9125 tablet_bootstrap.cc:654] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:49.867602 9127 tablet_bootstrap.cc:654] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:49.869264 9125 log.cc:824] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:49.869297 9127 log.cc:824] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:49.873158 9126 tablet_bootstrap.cc:654] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:49.874675 9126 log.cc:824] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:49.879175 9125 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: No bootstrap required, opened a new log
I20260928 16:46:49.879501 9125 ts_tablet_manager.cc:1397] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260928 16:46:49.880030 9127 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: No bootstrap required, opened a new log
I20260928 16:46:49.880187 9126 tablet_bootstrap.cc:492] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: No bootstrap required, opened a new log
I20260928 16:46:49.880357 9127 ts_tablet_manager.cc:1397] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260928 16:46:49.880443 9126 ts_tablet_manager.cc:1397] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20260928 16:46:49.888073 9125 raft_consensus.cc:348] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.888509 9125 raft_consensus.cc:374] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:49.888636 9125 raft_consensus.cc:729] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a760c3939744e3f906f1b6245a65672, State: Initialized, Role: FOLLOWER
I20260928 16:46:49.889283 9125 consensus_queue.cc:260] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.890472 8973 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:49.891327 9125 ts_tablet_manager.cc:1428] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 16:46:49.891414 9126 raft_consensus.cc:348] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.891417 9127 raft_consensus.cc:348] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.891912 9126 raft_consensus.cc:374] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:49.891916 9127 raft_consensus.cc:374] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:49.892031 9127 raft_consensus.cc:729] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc027b7b675497cb9f774ff38377a74, State: Initialized, Role: FOLLOWER
I20260928 16:46:49.892040 9126 raft_consensus.cc:729] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bf48241ac24dabb59f53e88744bb09, State: Initialized, Role: FOLLOWER
I20260928 16:46:49.892742 9127 consensus_queue.cc:260] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.892742 9126 consensus_queue.cc:260] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:49.894426 9106 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:49.895172 9126 ts_tablet_manager.cc:1428] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 16:46:49.896322 9127 ts_tablet_manager.cc:1428] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260928 16:46:49.939815 8839 tablet.cc:2369] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:46:49.955241 8974 tablet.cc:2369] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:46:49.970041 9107 tablet.cc:2369] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:50.258024 9132 raft_consensus.cc:482] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:46:50.258404 9132 raft_consensus.cc:504] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:50.260243 9132 leader_election.cc:288] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827), 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
I20260928 16:46:50.267217 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.260724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:38971 (local address 127.26.101.3:39827)
0928 16:46:50.260960 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.260969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.261564 (+ 595us) server_negotiation.cc:407] Connection header received
0928 16:46:50.261874 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.261881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.261965 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.262059 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:50.262896 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.263665 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.265459 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.266233 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.266386 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.266462 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.266774 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.266914 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:46:50.267354 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.260672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:39827 (local address 127.26.101.1:38971)
0928 16:46:50.261429 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:50.261452 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:50.261711 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:50.262237 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:50.262247 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:50.262276 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:50.262713 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:50.262728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.263804 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.263810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:50.265290 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:50.265304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.266458 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.266464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:50.266609 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.266633 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:46:50.266732 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:46:50.266941 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"spinlock_wait_cycles":26496,"thread_start_us":405,"threads_started":1}
I20260928 16:46:50.267489 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.261987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:42441 (local address 127.26.101.2:34101)
0928 16:46:50.262179 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.262185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.262356 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:46:50.262652 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.262657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.262728 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.262812 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:50.263362 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.264071 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.265767 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.266557 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.266719 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.266793 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.267124 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.267254 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:46:50.267763 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.261591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34101 (local address 127.26.101.1:42441)
0928 16:46:50.262261 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:50.262278 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:50.262487 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:50.262849 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:50.262854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:50.262880 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:50.263195 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:50.263205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.264207 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.264211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:50.265604 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:50.265618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.266776 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.266782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:50.266948 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.266975 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:46:50.267060 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:46:50.267223 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":2,"spinlock_wait_cycles":6272,"thread_start_us":571,"threads_started":1}
I20260928 16:46:50.268589 9061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce226ef09b14e7da3a85cd857e57e11" candidate_uuid: "ecc027b7b675497cb9f774ff38377a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6bf48241ac24dabb59f53e88744bb09" is_pre_election: true
I20260928 16:46:50.268605 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce226ef09b14e7da3a85cd857e57e11" candidate_uuid: "ecc027b7b675497cb9f774ff38377a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a760c3939744e3f906f1b6245a65672" is_pre_election: true
I20260928 16:46:50.269089 9061 raft_consensus.cc:2454] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc027b7b675497cb9f774ff38377a74 in term 0.
I20260928 16:46:50.269131 8927 raft_consensus.cc:2454] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc027b7b675497cb9f774ff38377a74 in term 0.
I20260928 16:46:50.269681 8725 leader_election.cc:302] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6bf48241ac24dabb59f53e88744bb09, ecc027b7b675497cb9f774ff38377a74; no voters:
I20260928 16:46:50.270083 9132 raft_consensus.cc:2781] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:46:50.270251 9132 raft_consensus.cc:482] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:46:50.270364 9132 raft_consensus.cc:3037] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:50.272620 9132 raft_consensus.cc:504] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:50.273252 9132 leader_election.cc:288] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [CANDIDATE]: Term 1 election: Requested vote from peers b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827), 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
I20260928 16:46:50.273902 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce226ef09b14e7da3a85cd857e57e11" candidate_uuid: "ecc027b7b675497cb9f774ff38377a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a760c3939744e3f906f1b6245a65672"
I20260928 16:46:50.273902 9061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce226ef09b14e7da3a85cd857e57e11" candidate_uuid: "ecc027b7b675497cb9f774ff38377a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6bf48241ac24dabb59f53e88744bb09"
I20260928 16:46:50.274132 9061 raft_consensus.cc:3037] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:50.274191 8927 raft_consensus.cc:3037] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:50.277419 8927 raft_consensus.cc:2454] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc027b7b675497cb9f774ff38377a74 in term 1.
I20260928 16:46:50.277423 9061 raft_consensus.cc:2454] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc027b7b675497cb9f774ff38377a74 in term 1.
I20260928 16:46:50.277967 8725 leader_election.cc:302] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6bf48241ac24dabb59f53e88744bb09, ecc027b7b675497cb9f774ff38377a74; no voters:
I20260928 16:46:50.278311 9132 raft_consensus.cc:2781] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:50.278784 9132 raft_consensus.cc:686] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 1 LEADER]: Becoming Leader. State: Replica: ecc027b7b675497cb9f774ff38377a74, State: Running, Role: LEADER
I20260928 16:46:50.279469 9132 consensus_queue.cc:237] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } }
I20260928 16:46:50.284339 8643 catalog_manager.cc:5495] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc027b7b675497cb9f774ff38377a74 (127.26.101.1). New cstate: current_term: 1 leader_uuid: "ecc027b7b675497cb9f774ff38377a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20260928 16:46:50.385557 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.379641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.26.101.1:46871)
0928 16:46:50.380125 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.380130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.380155 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:46:50.380303 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.380307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.380403 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.380564 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:46:50.380925 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.381654 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.383193 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.383592 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.383683 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.384280 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:46:50.384873 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:46:50.384999 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.385204 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.385294 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260928 16:46:50.394420 8927 raft_consensus.cc:1261] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Refusing update from remote peer ecc027b7b675497cb9f774ff38377a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:50.394423 9061 raft_consensus.cc:1261] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Refusing update from remote peer ecc027b7b675497cb9f774ff38377a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:50.395535 9136 consensus_queue.cc:1035] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:50.396076 9132 consensus_queue.cc:1035] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:50.408442 9141 mvcc.cc:205] Tried to move back new op lower bound from 7334354986563739648 to 7334354986109677568. Current Snapshot: MvccSnapshot[applied={T|T < 7334354986563739648}]
I20260928 16:46:50.421878 8643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:50.422212 8643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:50.426309 8643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Sending DeleteTablet for 3 replicas of tablet cce226ef09b14e7da3a85cd857e57e11
I20260928 16:46:50.427663 9041 tablet_service.cc:1514] Processing DeleteTablet for tablet cce226ef09b14e7da3a85cd857e57e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:52806
I20260928 16:46:50.427666 8772 tablet_service.cc:1514] Processing DeleteTablet for tablet cce226ef09b14e7da3a85cd857e57e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:53436
I20260928 16:46:50.428020 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8708
I20260928 16:46:50.428969 9154 tablet_replica.cc:330] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: stopping tablet replica
I20260928 16:46:50.429450 9154 raft_consensus.cc:2229] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:46:50.429451 8907 tablet_service.cc:1514] Processing DeleteTablet for tablet cce226ef09b14e7da3a85cd857e57e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:50 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:34588
I20260928 16:46:50.430166 9154 raft_consensus.cc:2258] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:50.430980 9153 tablet_replica.cc:330] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: stopping tablet replica
I20260928 16:46:50.431512 9153 raft_consensus.cc:2229] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:50.432070 9153 raft_consensus.cc:2258] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:50.434034 9154 ts_tablet_manager.cc:1905] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:50.440654 9153 ts_tablet_manager.cc:1905] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:50.441606 9156 tablet_replica.cc:330] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: stopping tablet replica
Sep 28 16:46:50 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:46:50.442080 9156 raft_consensus.cc:2229] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:50.442615 9156 raft_consensus.cc:2258] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:50.443923 9154 ts_tablet_manager.cc:1918] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:50.444188 9154 log.cc:1196] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/cce226ef09b14e7da3a85cd857e57e11
I20260928 16:46:50.444797 9156 ts_tablet_manager.cc:1905] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:50.444976 9154 ts_tablet_manager.cc:1939] T cce226ef09b14e7da3a85cd857e57e11 P ecc027b7b675497cb9f774ff38377a74: Deleting consensus metadata
I20260928 16:46:50.447026 8630 catalog_manager.cc:4841] TS ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871): tablet cce226ef09b14e7da3a85cd857e57e11 (table test-table [id=5cd8f43779d7402da4434d4b39917a65]) successfully deleted
I20260928 16:46:50.447742 9153 ts_tablet_manager.cc:1918] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:50.447913 9153 log.cc:1196] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/cce226ef09b14e7da3a85cd857e57e11
I20260928 16:46:50.448427 9153 ts_tablet_manager.cc:1939] T cce226ef09b14e7da3a85cd857e57e11 P b6bf48241ac24dabb59f53e88744bb09: Deleting consensus metadata
I20260928 16:46:50.450363 8629 catalog_manager.cc:4841] TS b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827): tablet cce226ef09b14e7da3a85cd857e57e11 (table test-table [id=5cd8f43779d7402da4434d4b39917a65]) successfully deleted
I20260928 16:46:50.450629 9156 ts_tablet_manager.cc:1918] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:50.450790 9156 log.cc:1196] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/cce226ef09b14e7da3a85cd857e57e11
I20260928 16:46:50.451303 9156 ts_tablet_manager.cc:1939] T cce226ef09b14e7da3a85cd857e57e11 P 4a760c3939744e3f906f1b6245a65672: Deleting consensus metadata
I20260928 16:46:50.453094 8629 catalog_manager.cc:4841] TS 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101): tablet cce226ef09b14e7da3a85cd857e57e11 (table test-table [id=5cd8f43779d7402da4434d4b39917a65]) successfully deleted
I20260928 16:46:50.453754 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.428987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.26.101.1:46871)
0928 16:46:50.429143 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.429149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.429744 (+ 595us) server_negotiation.cc:407] Connection header received
0928 16:46:50.430051 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.430057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.430160 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.430239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:50.432596 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.433938 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.436243 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.437041 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.437163 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.442776 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:50.442793 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:50.442808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:50.442845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:50.447133 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.451149 (+ 4016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.451160 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.451163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.451356 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.451873 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.451878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.451881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.452428 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:50.452755 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.453143 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.453432 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:46:50.454484 8833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:50.551304 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8708
I20260928 16:46:50.568889 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8844
Sep 28 16:46:50 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:46:50.589229 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.569697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.26.101.2:34101)
0928 16:46:50.569892 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.569897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.569915 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:46:50.570076 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.570081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.570145 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.570226 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:50.571594 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.572358 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.573691 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.574106 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.574239 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.582889 (+ 8650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:50.582909 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:50.582925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:50.582969 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:50.586075 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.586948 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.586957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.586960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.587152 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.587563 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.587569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.587572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.588016 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:50.588313 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.588644 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.588894 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:46:50.589955 8967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:50.688463 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8844
I20260928 16:46:50.705040 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8978
Sep 28 16:46:50 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:46:50.721992 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:50.706326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.26.101.3:39827)
0928 16:46:50.706524 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:50.706531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:50.706555 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:46:50.706628 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:50.706634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:50.706722 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:50.706857 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:50.708435 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.709574 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.710907 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:50.711397 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:50.711517 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:50.716066 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:50.716077 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:50.716092 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:50.716142 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:50.719120 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.719983 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.719994 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.719997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.720199 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:50.720582 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:50.720588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:50.720590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:50.720949 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:50.721181 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:50.721490 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 16:46:50.721682 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:46:50.722739 9101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:50.813354 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8978
I20260928 16:46:50.828795 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 8612
I20260928 16:46:50.830235 8676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:50.941090 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 8612
I20260928 16:46:50.961198 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--webserver_interface=127.26.101.62
--webserver_port=39509
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:51.304282 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:51.304630 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:51.304781 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:51.312345 9161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:51.312479 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:51.312532 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:51.312613 9161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:51.312670 9161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:51.323159 9161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:51.324604 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:51.326120 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:51.352003 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:51.352046 9168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:51.352067 9167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:51.352991 9161 server_base.cc:1034] running on GCE node
I20260928 16:46:51.354025 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:51.355827 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:51.357146 9161 hybrid_clock.cc:648] HybridClock initialized: now 1790614011357072 us; error 48 us; skew 500 ppm
I20260928 16:46:51.357666 9161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:51 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614011, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:51.361420 9161 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:51.363201 9161 webserver.cc:460] Webserver started at http://127.26.101.62:39509/ using document root <none> and password file <none>
I20260928 16:46:51.364044 9161 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:51.364188 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:51.370602 9161 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:46:51.373785 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:51.375425 9161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:51.375620 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "3e1b5cf2cc164c1ea332a80b4398d181"
format_stamp: "Formatted at 2026-09-28 16:46:47 on dist-test-slave-t0tw"
server_key: "2206856fe301298583f4f132b5b3f5aa"
server_key_iv: "1bdfc11d256814949ac1b737440069bc"
server_key_version: "encryptionkey@0"
I20260928 16:46:51.376736 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:51.408052 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:51.412801 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:51.413091 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:51.432528 9161 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45955
I20260928 16:46:51.432545 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45955 every 8 connection(s)
I20260928 16:46:51.434191 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:51.435468 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9161
I20260928 16:46:51.436273 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:46871
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=33001
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:51.441946 9229 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:46:51.445782 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap starting.
I20260928 16:46:51.466621 9229 log.cc:824] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:51.479211 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:46:51.479969 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap complete.
I20260928 16:46:51.489663 9229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:51.490569 9229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Initialized, Role: FOLLOWER
I20260928 16:46:51.491274 9229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:51.491523 9229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:51.491695 9229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:51.491864 9229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:46:51.495958 9229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:51.496510 9229 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1b5cf2cc164c1ea332a80b4398d181; no voters:
I20260928 16:46:51.497144 9229 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:46:51.497296 9233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:46:51.498454 9233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 LEADER]: Becoming Leader. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Running, Role: LEADER
I20260928 16:46:51.499169 9233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:51.499670 9229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:51.501714 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1b5cf2cc164c1ea332a80b4398d181. Latest consensus state: current_term: 2 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:51.502475 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:51.502134 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:51.503049 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:51.512742 9239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:51.522186 9239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:51.523846 9239 catalog_manager.cc:1259] Loaded cluster ID: 9110b440f3bd49a28a5e51665b97871c
I20260928 16:46:51.524051 9239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:46:51.527266 9239 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 16:46:51.544469 9239 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 16:46:51.555279 9239 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:46:51.557207 9239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Loaded TSK: 0
I20260928 16:46:51.558820 9239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:46:51.795018 9231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:51.795397 9231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:51.795540 9231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:51.803681 9231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:51.803809 9231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:51.803956 9231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:46:51.814610 9231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:46871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=33001
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:51.816026 9231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:51.817411 9231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:51.833150 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:51.832950 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:51.833433 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:51.834156 9231 server_base.cc:1034] running on GCE node
I20260928 16:46:51.834812 9231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:51.835930 9231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:51.837162 9231 hybrid_clock.cc:648] HybridClock initialized: now 1790614011837090 us; error 62 us; skew 500 ppm
I20260928 16:46:51.837491 9231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:51 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614011, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:51.841110 9231 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:46:51.842808 9231 webserver.cc:460] Webserver started at http://127.26.101.1:33001/ using document root <none> and password file <none>
I20260928 16:46:51.843616 9231 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:51.843761 9231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:51.849606 9231 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:46:51.852597 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:51.854089 9231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:51.854265 9231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "ecc027b7b675497cb9f774ff38377a74"
format_stamp: "Formatted at 2026-09-28 16:46:48 on dist-test-slave-t0tw"
server_key: "311cad86112bcd78342482ff7bb322f4"
server_key_iv: "a89217e3a3f34cb0d1119fa162a78fad"
server_key_version: "encryptionkey@0"
I20260928 16:46:51.855377 9231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:51.868465 9231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:51.873178 9231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:51.873461 9231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:51.874567 9231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:51.876130 9231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:51.876223 9231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:51.876328 9231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:51.876395 9231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:51.911896 9231 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:46871
I20260928 16:46:51.911963 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:46871 every 8 connection(s)
I20260928 16:46:51.913547 9231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:51.920145 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9231
I20260928 16:46:51.920919 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:34101
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=39423
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:51 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614011, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:51.942781 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:51.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:59963 (local address 127.26.101.62:45955)
0928 16:46:51.917486 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:51.917499 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:51.918349 (+ 850us) server_negotiation.cc:407] Connection header received
0928 16:46:51.920803 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:51.920829 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:51.921376 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:51.921858 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:51.923981 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:51.925902 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:51.927791 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:51.928766 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:51.928919 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:51.933629 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:51.933638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:51.933658 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:51.933714 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:51.937125 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:51.938358 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:51.938372 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:51.938381 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:51.938652 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:51.939228 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:51.939232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:51.939234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:51.939779 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:51.940022 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:51.940988 (+ 966us) server_negotiation.cc:299] Negotiation successful
0928 16:46:51.941454 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
I20260928 16:46:51.942781 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:51.917169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:59963)
0928 16:46:51.918098 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:51.918156 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:51.920043 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:51.922070 (+ 2027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:51.922079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:51.922921 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:51.923662 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:51.923685 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:51.926177 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:51.926182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:51.927619 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:51.927637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:51.929116 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:51.929126 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:51.929429 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:51.930277 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:51.930307 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:51.933197 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:51.937371 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:51.937383 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:51.937398 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:51.937986 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:51.938796 (+ 810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:51.938802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:51.938806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:51.939059 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:51.939951 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:51.939962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:51.940363 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 16:46:51.940693 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 16:46:51.941372 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":361,"threads_started":1}
I20260928 16:46:51.946429 9380 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:51.947275 9380 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:51.948482 9380 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:51.951620 9194 ts_manager.cc:194] Registered new tserver with Master: ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871)
I20260928 16:46:51.953341 9194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:59963
W20260928 16:46:52.269680 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:52.270030 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:52.270152 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:52.277595 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:52.277731 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:52.277877 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:46:52.287993 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:34101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=39423
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:52.289399 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:52.290798 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:52.304924 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:52.304951 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:52.304862 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:52.305521 9384 server_base.cc:1034] running on GCE node
I20260928 16:46:52.306200 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:52.307237 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:52.308473 9384 hybrid_clock.cc:648] HybridClock initialized: now 1790614012308426 us; error 38 us; skew 500 ppm
I20260928 16:46:52.308849 9384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:52 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614012, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:52.312389 9384 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:46:52.314116 9384 webserver.cc:460] Webserver started at http://127.26.101.2:39423/ using document root <none> and password file <none>
I20260928 16:46:52.314904 9384 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:52.315040 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:52.321033 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:46:52.324223 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.325937 9384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:46:52.326159 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "4a760c3939744e3f906f1b6245a65672"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "e7fcb73c8607b90a152771cef31ee45d"
server_key_iv: "8c35bbcde8b3f0b8b5466c00220cdaf5"
server_key_version: "encryptionkey@0"
I20260928 16:46:52.327278 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:52.349313 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:52.353865 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:52.354206 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:52.355315 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:52.356709 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:52.356808 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.356925 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:52.357007 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.388887 9384 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:34101
I20260928 16:46:52.388947 9514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:34101 every 8 connection(s)
I20260928 16:46:52.390309 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:52.397737 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9384
I20260928 16:46:52.398484 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:39827
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=39301
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:46:52 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614012, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:52.413640 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.392869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:45951 (local address 127.26.101.62:45955)
0928 16:46:52.393041 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.393046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.394278 (+ 1232us) server_negotiation.cc:407] Connection header received
0928 16:46:52.396315 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.396321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.396409 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.396498 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:52.398570 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.399575 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.401888 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.402437 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.402550 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.407025 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:52.407032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:52.407037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:52.407115 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:52.410287 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.411172 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.411178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.411182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.411338 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.411773 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.411778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.411780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.412134 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:52.412279 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.413033 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.413296 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:46:52.414584 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.393374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:45951)
0928 16:46:52.394110 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:52.394150 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:52.396013 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:52.396778 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:52.396787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:52.397421 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:52.398327 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.398355 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.399770 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.399774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.401714 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.401733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.402709 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.402717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.403005 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.403833 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:52.403868 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:52.406724 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:52.410458 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:52.410466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:52.410481 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:52.410966 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:52.411446 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:52.411449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:52.411452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:52.411664 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:52.412274 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:52.412280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:52.412630 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 16:46:52.412898 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 16:46:52.413373 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
I20260928 16:46:52.416074 9515 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:52.416445 9515 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:52.417402 9515 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:52.419145 9194 ts_manager.cc:194] Registered new tserver with Master: 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
I20260928 16:46:52.419997 9194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:45951
W20260928 16:46:52.742431 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:52.742729 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:52.742862 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:52.750406 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:52.750521 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:52.750667 9518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:46:52.760617 9518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:39827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=39301
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:52.761894 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:52.763285 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:52.777247 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:52.777262 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:52.778003 9518 server_base.cc:1034] running on GCE node
W20260928 16:46:52.778064 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:52.778719 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:52.779771 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:52.780969 9518 hybrid_clock.cc:648] HybridClock initialized: now 1790614012780918 us; error 47 us; skew 500 ppm
I20260928 16:46:52.781293 9518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:52 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614012, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:52.784368 9518 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:46:52.785804 9518 webserver.cc:460] Webserver started at http://127.26.101.3:39301/ using document root <none> and password file <none>
I20260928 16:46:52.786481 9518 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:52.786619 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:52.791937 9518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:46:52.794574 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.795985 9518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:46:52.796141 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "b6bf48241ac24dabb59f53e88744bb09"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "93749f0f595a7384b88425a9589d45f4"
server_key_iv: "f53ee6f05c6c314d625327aabe9cbfcd"
server_key_version: "encryptionkey@0"
I20260928 16:46:52.797106 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:52.811058 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:52.815152 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:52.815404 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:52.816380 9518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:52.817685 9518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:52.817785 9518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.817891 9518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:52.817960 9518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:52.848656 9518 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:39827
I20260928 16:46:52.848737 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:39827 every 8 connection(s)
I20260928 16:46:52.850203 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:52.856376 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9518
Sep 28 16:46:52 dist-test-slave-t0tw krb5kdc[8599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614012, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:46:52.873678 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.857369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.26.101.62:45955)
0928 16:46:52.857891 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.857897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.858365 (+ 468us) server_negotiation.cc:407] Connection header received
0928 16:46:52.858491 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.858496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.858577 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.858697 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:52.860244 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.861279 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.862936 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.863435 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.863546 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.866503 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:52.866511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:52.866515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:52.866558 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:52.870343 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.871193 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.871199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.871202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.871415 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.871897 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.871903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.871907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.872391 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:52.872640 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.873036 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.873239 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":2432,"thread_start_us":236,"threads_started":1}
I20260928 16:46:52.874008 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.852868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:59237 (local address 127.26.101.62:45955)
0928 16:46:52.853096 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.853102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.854264 (+ 1162us) server_negotiation.cc:407] Connection header received
0928 16:46:52.856187 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.856191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.856262 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.856349 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:52.858225 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.859279 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.861306 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.861922 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.862052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.866601 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:52.866606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:52.866610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:52.866655 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:52.870310 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.871237 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.871242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.871245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.871423 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:52.871958 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:52.871967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:52.871970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:52.872383 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:52.872560 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.873432 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.873670 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":8576}
I20260928 16:46:52.875198 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.853336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:59237)
0928 16:46:52.854099 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:52.854144 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:52.855866 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:52.856644 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:52.856656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:52.857284 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:46:52.858054 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.858073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.859433 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.859437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.861155 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.861176 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.862165 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.862170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.862386 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.863796 (+ 1410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:46:52.863821 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:46:52.866362 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:46:52.870478 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:52.870489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:52.870507 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:52.871144 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:52.871549 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:46:52.871553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:46:52.871556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:46:52.871827 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:46:52.872516 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:46:52.872523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:46:52.872969 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 16:46:52.873296 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 16:46:52.873801 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":284,"threads_started":1}
I20260928 16:46:52.876658 9647 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45955
I20260928 16:46:52.876967 9647 heartbeater.cc:464] Registering TS with master...
I20260928 16:46:52.877696 9647 heartbeater.cc:511] Master 127.26.101.62:45955 requested a full tablet report, sending...
I20260928 16:46:52.879217 9194 ts_manager.cc:194] Registered new tserver with Master: b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827)
I20260928 16:46:52.880045 9194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:59237
I20260928 16:46:52.889170 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:46:52.896123 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.890144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.26.101.62:45955)
0928 16:46:52.890314 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.890319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.890726 (+ 407us) server_negotiation.cc:407] Connection header received
0928 16:46:52.890925 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.890929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.891068 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.891235 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:46:52.891672 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.892416 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.893984 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.894425 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.894517 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.895041 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:46:52.895613 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:46:52.895724 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.895843 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.895917 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:46:52.897039 9194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:46:52.904652 9194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:46:52.929234 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.917802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34101 (local address 127.0.0.1:34602)
0928 16:46:52.921201 (+ 3399us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:52.921223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:52.921341 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:52.922005 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:52.922010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:52.922040 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:52.922475 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.922488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.924311 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.924316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.926806 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.926820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.928200 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.928206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.928393 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.928448 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 16:46:52.928606 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 16:46:52.928801 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3278,"mutex_wait_us":147,"thread_start_us":3124,"threads_started":1}
I20260928 16:46:52.929513 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.918226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.26.101.2:34101)
0928 16:46:52.919584 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.919596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.921467 (+ 1871us) server_negotiation.cc:407] Connection header received
0928 16:46:52.921594 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.921601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.921816 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.921984 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:52.922710 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.924107 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.926964 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.927970 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.928117 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.928332 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.929017 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.929219 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":1021,"threads_started":1}
I20260928 16:46:52.930787 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.917669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:46871 (local address 127.0.0.1:53452)
0928 16:46:52.920529 (+ 2860us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:52.920579 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:52.920804 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:52.921857 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:52.921865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:52.921896 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:52.922409 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.922429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.924382 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.924387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.926821 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.926836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.929909 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.929914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.930076 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.930118 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:46:52.930214 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:46:52.930356 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"mutex_wait_us":144,"thread_start_us":2452,"threads_started":1}
I20260928 16:46:52.930792 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.918133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.26.101.1:46871)
0928 16:46:52.920089 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.920098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.920727 (+ 629us) server_negotiation.cc:407] Connection header received
0928 16:46:52.921114 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.921123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.921381 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.921572 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:52.922660 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.924192 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.927025 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.928494 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.928630 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.928855 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.930291 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.930470 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1781,"thread_start_us":1648,"threads_started":1}
I20260928 16:46:52.931733 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.921907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.26.101.3:39827)
0928 16:46:52.922647 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:52.922656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:52.923207 (+ 551us) server_negotiation.cc:407] Connection header received
0928 16:46:52.923367 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:52.923376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:52.923584 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:52.923781 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:52.924361 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.926057 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.928513 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.929870 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.929985 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.930213 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:52.931266 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 16:46:52.931453 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":411,"threads_started":1}
I20260928 16:46:52.932029 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:52.921350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:39827 (local address 127.0.0.1:52828)
0928 16:46:52.923100 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:52.923120 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:52.923237 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:52.923780 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:52.923786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:52.923812 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:52.924174 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.924187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.926215 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.926220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.928327 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:52.928342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:52.930374 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:52.930380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:52.930568 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:52.930602 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:46:52.930701 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 16:46:52.930838 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1597,"thread_start_us":1466,"threads_started":1}
I20260928 16:46:52.933559 9449 tablet_service.cc:1467] Processing CreateTablet for tablet f413ba69e9d7476db0f12a1fc718977b (DEFAULT_TABLE table=test-table [id=c72eb980f596484f8deb9c5cfac23e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:52.935266 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f413ba69e9d7476db0f12a1fc718977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:52.935424 9315 tablet_service.cc:1467] Processing CreateTablet for tablet f413ba69e9d7476db0f12a1fc718977b (DEFAULT_TABLE table=test-table [id=c72eb980f596484f8deb9c5cfac23e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:52.935731 9582 tablet_service.cc:1467] Processing CreateTablet for tablet f413ba69e9d7476db0f12a1fc718977b (DEFAULT_TABLE table=test-table [id=c72eb980f596484f8deb9c5cfac23e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:46:52.936618 9315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f413ba69e9d7476db0f12a1fc718977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:52.937072 9582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f413ba69e9d7476db0f12a1fc718977b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:46:52.948946 9663 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Bootstrap starting.
I20260928 16:46:52.952239 9664 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Bootstrap starting.
I20260928 16:46:52.952678 9663 tablet_bootstrap.cc:654] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:52.954073 9665 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Bootstrap starting.
I20260928 16:46:52.955806 9663 log.cc:824] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:52.956297 9664 tablet_bootstrap.cc:654] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:52.957525 9380 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:52.957695 9665 tablet_bootstrap.cc:654] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Neither blocks nor log segments found. Creating new log.
I20260928 16:46:52.957690 9664 log.cc:824] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:52.958990 9665 log.cc:824] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:52.959640 9663 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: No bootstrap required, opened a new log
I20260928 16:46:52.960017 9663 ts_tablet_manager.cc:1397] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 16:46:52.965646 9665 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: No bootstrap required, opened a new log
I20260928 16:46:52.966017 9665 ts_tablet_manager.cc:1397] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260928 16:46:52.967551 9664 tablet_bootstrap.cc:492] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: No bootstrap required, opened a new log
I20260928 16:46:52.967813 9664 ts_tablet_manager.cc:1397] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260928 16:46:52.970783 9663 raft_consensus.cc:348] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.971361 9663 raft_consensus.cc:374] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:52.971510 9663 raft_consensus.cc:729] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc027b7b675497cb9f774ff38377a74, State: Initialized, Role: FOLLOWER
I20260928 16:46:52.972187 9663 consensus_queue.cc:260] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.974038 9663 ts_tablet_manager.cc:1428] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 16:46:52.976336 9665 raft_consensus.cc:348] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.976500 9664 raft_consensus.cc:348] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.976824 9665 raft_consensus.cc:374] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:52.976923 9664 raft_consensus.cc:374] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:46:52.976929 9665 raft_consensus.cc:729] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bf48241ac24dabb59f53e88744bb09, State: Initialized, Role: FOLLOWER
I20260928 16:46:52.977042 9664 raft_consensus.cc:729] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a760c3939744e3f906f1b6245a65672, State: Initialized, Role: FOLLOWER
I20260928 16:46:52.977533 9665 consensus_queue.cc:260] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.977782 9664 consensus_queue.cc:260] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:52.978679 9647 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:52.979375 9665 ts_tablet_manager.cc:1428] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 16:46:52.979691 9515 heartbeater.cc:503] Master 127.26.101.62:45955 was elected leader, sending a full tablet report...
I20260928 16:46:52.979697 9664 ts_tablet_manager.cc:1428] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
W20260928 16:46:53.101861 9648 tablet.cc:2369] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:53.120429 9670 raft_consensus.cc:482] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:46:53.120741 9670 raft_consensus.cc:504] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:53.122570 9670 leader_election.cc:288] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871), 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
I20260928 16:46:53.129292 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.123100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:36751 (local address 127.26.101.1:46871)
0928 16:46:53.123308 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.123314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.123395 (+ 81us) server_negotiation.cc:407] Connection header received
0928 16:46:53.123654 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.123661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.123746 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.123860 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:53.124262 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.125232 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.127177 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.128341 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.128474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.128531 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.128916 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.129074 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:46:53.129294 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.122937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:46871 (local address 127.26.101.3:36751)
0928 16:46:53.123274 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:53.123299 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:53.123457 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:53.123891 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:53.123895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:53.123920 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:53.124120 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:53.124132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.125360 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.125366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:53.126962 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:53.126977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.128517 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.128524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:53.128717 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.128754 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:46:53.128857 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:46:53.129002 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260928 16:46:53.130401 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.123211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:53431 (local address 127.26.101.2:34101)
0928 16:46:53.123420 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.123426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.124069 (+ 643us) server_negotiation.cc:407] Connection header received
0928 16:46:53.124157 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.124162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.124237 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.124309 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:46:53.124929 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.126053 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.128264 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.129412 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.129563 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.129636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.130098 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.130210 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:46:53.130474 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.122991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:34101 (local address 127.26.101.3:53431)
0928 16:46:53.123899 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:53.123917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:46:53.124042 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:46:53.124345 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:46:53.124349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:46:53.124374 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:46:53.124735 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:53.124754 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.126230 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.126242 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:53.128063 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:46:53.128081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.129706 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.129712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:46:53.129855 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.129882 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:46:53.130007 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 16:46:53.130147 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":470,"threads_started":1}
I20260928 16:46:53.130473 9335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f413ba69e9d7476db0f12a1fc718977b" candidate_uuid: "b6bf48241ac24dabb59f53e88744bb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc027b7b675497cb9f774ff38377a74" is_pre_election: true
I20260928 16:46:53.131201 9335 raft_consensus.cc:2454] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bf48241ac24dabb59f53e88744bb09 in term 0.
I20260928 16:46:53.131583 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f413ba69e9d7476db0f12a1fc718977b" candidate_uuid: "b6bf48241ac24dabb59f53e88744bb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a760c3939744e3f906f1b6245a65672" is_pre_election: true
I20260928 16:46:53.131884 9536 leader_election.cc:302] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6bf48241ac24dabb59f53e88744bb09, ecc027b7b675497cb9f774ff38377a74; no voters:
I20260928 16:46:53.132145 9469 raft_consensus.cc:2454] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bf48241ac24dabb59f53e88744bb09 in term 0.
I20260928 16:46:53.132334 9670 raft_consensus.cc:2781] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:46:53.132474 9670 raft_consensus.cc:482] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:46:53.132599 9670 raft_consensus.cc:3037] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:53.134878 9670 raft_consensus.cc:504] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
I20260928 16:46:53.135648 9670 leader_election.cc:288] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [CANDIDATE]: Term 1 election: Requested vote from peers ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871), 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101)
I20260928 16:46:53.136219 9335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f413ba69e9d7476db0f12a1fc718977b" candidate_uuid: "b6bf48241ac24dabb59f53e88744bb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc027b7b675497cb9f774ff38377a74"
I20260928 16:46:53.136265 9469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f413ba69e9d7476db0f12a1fc718977b" candidate_uuid: "b6bf48241ac24dabb59f53e88744bb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a760c3939744e3f906f1b6245a65672"
I20260928 16:46:53.136427 9335 raft_consensus.cc:3037] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:53.136474 9469 raft_consensus.cc:3037] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:46:53.138777 9335 raft_consensus.cc:2454] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bf48241ac24dabb59f53e88744bb09 in term 1.
I20260928 16:46:53.138827 9469 raft_consensus.cc:2454] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bf48241ac24dabb59f53e88744bb09 in term 1.
I20260928 16:46:53.139336 9536 leader_election.cc:302] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6bf48241ac24dabb59f53e88744bb09, ecc027b7b675497cb9f774ff38377a74; no voters:
I20260928 16:46:53.139673 9670 raft_consensus.cc:2781] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:46:53.140167 9670 raft_consensus.cc:686] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 1 LEADER]: Becoming Leader. State: Replica: b6bf48241ac24dabb59f53e88744bb09, State: Running, Role: LEADER
I20260928 16:46:53.140703 9670 consensus_queue.cc:237] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } }
W20260928 16:46:53.142275 9516 tablet.cc:2369] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:53.145401 9193 catalog_manager.cc:5495] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "b6bf48241ac24dabb59f53e88744bb09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6bf48241ac24dabb59f53e88744bb09" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39827 } health_report { overall_health: HEALTHY } } }
W20260928 16:46:53.166069 9381 tablet.cc:2369] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:46:53.178422 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.172603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.26.101.3:39827)
0928 16:46:53.172814 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.172819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.172891 (+ 72us) server_negotiation.cc:407] Connection header received
0928 16:46:53.173096 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.173101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.173185 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.173311 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:46:53.173654 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.174413 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.176020 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.176470 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.176586 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.177264 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:46:53.177792 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:46:53.177907 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.178048 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.178140 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:46:53.187759 9335 raft_consensus.cc:1261] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Refusing update from remote peer b6bf48241ac24dabb59f53e88744bb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:53.187759 9469 raft_consensus.cc:1261] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Refusing update from remote peer b6bf48241ac24dabb59f53e88744bb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:46:53.188845 9673 consensus_queue.cc:1035] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc027b7b675497cb9f774ff38377a74" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:53.189210 9670 consensus_queue.cc:1035] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a760c3939744e3f906f1b6245a65672" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:46:53.201570 9677 mvcc.cc:205] Tried to move back new op lower bound from 7334354998004277248 to 7334354997828313088. Current Snapshot: MvccSnapshot[applied={T|T < 7334354998004277248}]
I20260928 16:46:53.222940 9193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:53.223336 9193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:46:53.226673 9193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Sending DeleteTablet for 3 replicas of tablet f413ba69e9d7476db0f12a1fc718977b
I20260928 16:46:53.228051 9315 tablet_service.cc:1514] Processing DeleteTablet for tablet f413ba69e9d7476db0f12a1fc718977b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:53 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:53452
I20260928 16:46:53.228061 9582 tablet_service.cc:1514] Processing DeleteTablet for tablet f413ba69e9d7476db0f12a1fc718977b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:53 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:52828
I20260928 16:46:53.228070 9449 tablet_service.cc:1514] Processing DeleteTablet for tablet f413ba69e9d7476db0f12a1fc718977b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:46:53 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:34602
I20260928 16:46:53.230549 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9231
I20260928 16:46:53.242740 9686 tablet_replica.cc:330] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: stopping tablet replica
I20260928 16:46:53.243316 9686 raft_consensus.cc:2229] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:53.243650 9687 tablet_replica.cc:330] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: stopping tablet replica
I20260928 16:46:53.244036 9685 tablet_replica.cc:330] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: stopping tablet replica
I20260928 16:46:53.244201 9687 raft_consensus.cc:2229] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:46:53.244612 9685 raft_consensus.cc:2229] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:46:53.244837 9687 raft_consensus.cc:2258] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:53.245400 9685 raft_consensus.cc:2258] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:53.245991 9686 raft_consensus.cc:2258] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:46:53.247200 9687 ts_tablet_manager.cc:1905] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:53.248695 9686 ts_tablet_manager.cc:1905] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:53.253039 9685 ts_tablet_manager.cc:1905] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:46:53.254256 9687 ts_tablet_manager.cc:1918] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:53.254521 9687 log.cc:1196] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/f413ba69e9d7476db0f12a1fc718977b
I20260928 16:46:53.255226 9687 ts_tablet_manager.cc:1939] T f413ba69e9d7476db0f12a1fc718977b P 4a760c3939744e3f906f1b6245a65672: Deleting consensus metadata
I20260928 16:46:53.256204 9686 ts_tablet_manager.cc:1918] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:53.256479 9686 log.cc:1196] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/f413ba69e9d7476db0f12a1fc718977b
I20260928 16:46:53.256253 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.231615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.26.101.1:46871)
0928 16:46:53.231814 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.231821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.231848 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:46:53.232052 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.232058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.232148 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.232264 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:53.234484 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.235878 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.238149 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.238883 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.239037 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.243587 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:53.243606 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:53.243626 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:53.243671 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:53.248142 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.249210 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.249219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.249223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.249410 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.249905 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.249911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.249915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.250436 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:53.255333 (+ 4897us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.255551 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.255839 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:46:53.257100 9686 ts_tablet_manager.cc:1939] T f413ba69e9d7476db0f12a1fc718977b P ecc027b7b675497cb9f774ff38377a74: Deleting consensus metadata
I20260928 16:46:53.257347 9179 catalog_manager.cc:4841] TS 4a760c3939744e3f906f1b6245a65672 (127.26.101.2:34101): tablet f413ba69e9d7476db0f12a1fc718977b (table test-table [id=c72eb980f596484f8deb9c5cfac23e42]) successfully deleted
I20260928 16:46:53.257903 9375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:53.265661 9685 ts_tablet_manager.cc:1918] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:46:53.265897 9685 log.cc:1196] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/f413ba69e9d7476db0f12a1fc718977b
I20260928 16:46:53.267374 9685 ts_tablet_manager.cc:1939] T f413ba69e9d7476db0f12a1fc718977b P b6bf48241ac24dabb59f53e88744bb09: Deleting consensus metadata
I20260928 16:46:53.270170 9179 catalog_manager.cc:4841] TS b6bf48241ac24dabb59f53e88744bb09 (127.26.101.3:39827): tablet f413ba69e9d7476db0f12a1fc718977b (table test-table [id=c72eb980f596484f8deb9c5cfac23e42]) successfully deleted
I20260928 16:46:53.354751 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9231
I20260928 16:46:53.357115 9180 catalog_manager.cc:4841] TS ecc027b7b675497cb9f774ff38377a74 (127.26.101.1:46871): tablet f413ba69e9d7476db0f12a1fc718977b (table test-table [id=c72eb980f596484f8deb9c5cfac23e42]) successfully deleted
I20260928 16:46:53.378975 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9384
I20260928 16:46:53.395885 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.379840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.26.101.2:34101)
0928 16:46:53.380039 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.380045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.380067 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:46:53.380135 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.380141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.380223 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.380352 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:53.381761 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.382941 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.384461 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.385116 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.385250 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.388007 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:53.388030 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:53.388051 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:53.388111 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:53.392308 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.393267 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.393278 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.393282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.393443 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.393875 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.393880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.393884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.394418 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:53.394737 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.395223 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.395493 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:46:53.396854 9510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:53.488631 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9384
I20260928 16:46:53.511107 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9518
I20260928 16:46:53.528369 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:53.511921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.26.101.3:39827)
0928 16:46:53.512110 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:46:53.512118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:46:53.512374 (+ 256us) server_negotiation.cc:407] Connection header received
0928 16:46:53.512578 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:46:53.512582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:46:53.512673 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:46:53.512779 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:46:53.514668 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.515641 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.517629 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:46:53.518129 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:46:53.518245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:46:53.521785 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:46:53.521798 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:46:53.521813 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:46:53.521862 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:46:53.525068 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.526108 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.526116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.526118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.526313 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:46:53.526753 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:46:53.526756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:46:53.526759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:46:53.527214 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:46:53.527475 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 16:46:53.527842 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 16:46:53.528028 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:46:53.529167 9642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:53.652020 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9518
I20260928 16:46:53.670465 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9161
I20260928 16:46:53.671633 9224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:46:53.788192 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9161
I20260928 16:46:53.808367 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--webserver_interface=127.26.101.62
--webserver_port=39509
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:46:54.147533 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:54.147881 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:54.148023 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:54.156085 9692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:46:54.156234 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:54.156312 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:54.156378 9692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:46:54.156435 9692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:46:54.168730 9692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=39509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:54.170192 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:54.171656 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:54.183928 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:54.183969 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:54.184175 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:54.185153 9692 server_base.cc:1034] running on GCE node
I20260928 16:46:54.185976 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:54.187598 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:54.188813 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790614014188747 us; error 57 us; skew 500 ppm
I20260928 16:46:54.189208 9692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:54 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614014, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:54.192541 9692 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:46:54.194170 9692 webserver.cc:460] Webserver started at http://127.26.101.62:39509/ using document root <none> and password file <none>
I20260928 16:46:54.194856 9692 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:54.194963 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:54.200685 9692 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:46:54.203908 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:54.205495 9692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:54.205682 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "3e1b5cf2cc164c1ea332a80b4398d181"
format_stamp: "Formatted at 2026-09-28 16:46:47 on dist-test-slave-t0tw"
server_key: "2206856fe301298583f4f132b5b3f5aa"
server_key_iv: "1bdfc11d256814949ac1b737440069bc"
server_key_version: "encryptionkey@0"
I20260928 16:46:54.206736 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:54.226397 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:54.230917 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:54.231213 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:54.249665 9692 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45955
I20260928 16:46:54.249692 9759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45955 every 8 connection(s)
I20260928 16:46:54.251291 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:46:54.252799 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9692
I20260928 16:46:54.253857 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:46871
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=33001
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:54.260748 9760 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:46:54.265054 9760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap starting.
I20260928 16:46:54.287351 9760 log.cc:824] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Log is configured to *not* fsync() on all Append() calls
I20260928 16:46:54.304352 9760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:46:54.304925 9760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181: Bootstrap complete.
I20260928 16:46:54.314488 9760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:54.315513 9760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Initialized, Role: FOLLOWER
I20260928 16:46:54.316174 9760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:54.316411 9760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:46:54.316524 9760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:46:54.316687 9760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:46:54.320694 9760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:54.321225 9760 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1b5cf2cc164c1ea332a80b4398d181; no voters:
I20260928 16:46:54.321880 9760 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:46:54.322031 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:46:54.323244 9764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [term 3 LEADER]: Becoming Leader. State: Replica: 3e1b5cf2cc164c1ea332a80b4398d181, State: Running, Role: LEADER
I20260928 16:46:54.323838 9764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } }
I20260928 16:46:54.324190 9760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:46:54.326453 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1b5cf2cc164c1ea332a80b4398d181. Latest consensus state: current_term: 3 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:54.326853 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:54.327715 9765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1b5cf2cc164c1ea332a80b4398d181" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45955 } } }
I20260928 16:46:54.328001 9765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1b5cf2cc164c1ea332a80b4398d181 [sys.catalog]: This master's current role is: LEADER
I20260928 16:46:54.334897 9771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:46:54.344287 9771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:46:54.345839 9771 catalog_manager.cc:1259] Loaded cluster ID: 9110b440f3bd49a28a5e51665b97871c
I20260928 16:46:54.345958 9771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 16:46:54.348119 9771 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd648d089ad google::LogMessage::Fail() at ??:0
@ 0x7fd648d0b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd648d084de google::LogMessage::Flush() at ??:0
@ 0x7fd648d09a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd65e73b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd65e738976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd65e7c0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd65e7c0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd64b954345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd64b95791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd64b957799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd64b932eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd64ec0c6db start_thread at ??:0
@ 0x7fd647ba261f clone at ??:0
W20260928 16:46:54.656451 9762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:54.656778 9762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:54.656909 9762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:54.664628 9762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:54.664757 9762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:54.664907 9762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:46:54.675686 9762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:46871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=33001
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:54.677096 9762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:54.678462 9762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:54.692660 9802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:54.692648 9801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:54.692608 9804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:54.693795 9762 server_base.cc:1034] running on GCE node
I20260928 16:46:54.694559 9762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:54.695786 9762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:54.697044 9762 hybrid_clock.cc:648] HybridClock initialized: now 1790614014696985 us; error 55 us; skew 500 ppm
I20260928 16:46:54.697413 9762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:54 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614014, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:54.701135 9762 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:46:54.702836 9762 webserver.cc:460] Webserver started at http://127.26.101.1:33001/ using document root <none> and password file <none>
I20260928 16:46:54.703689 9762 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:54.703804 9762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:54.709607 9762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:46:54.712769 9811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:54.714367 9762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:54.714571 9762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "ecc027b7b675497cb9f774ff38377a74"
format_stamp: "Formatted at 2026-09-28 16:46:48 on dist-test-slave-t0tw"
server_key: "311cad86112bcd78342482ff7bb322f4"
server_key_iv: "a89217e3a3f34cb0d1119fa162a78fad"
server_key_version: "encryptionkey@0"
I20260928 16:46:54.715715 9762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:54.729089 9762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:54.734160 9762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:54.734448 9762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:54.735605 9762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:54.737147 9762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:54.737243 9762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:54.737349 9762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:54.737417 9762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:54.770792 9762 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:46871
I20260928 16:46:54.770849 9924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:46871 every 8 connection(s)
I20260928 16:46:54.772361 9762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:46:54.772899 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9762
I20260928 16:46:54.773743 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:34101
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=39423
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:54.778365 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:54.775890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:40185)
0928 16:46:54.776785 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:54.777037 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":357,"threads_started":1}
W20260928 16:46:54.779439 9925 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:45955 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:45955: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
I20260928 16:46:54.780866 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:54.780388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:46837)
0928 16:46:54.780650 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:54.780738 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 16:46:54.781811 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:54.781443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:43447)
0928 16:46:54.781598 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:54.781686 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:46:54.782028 9925 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:46:55.121076 9928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:55.121410 9928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:55.121551 9928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:55.129024 9928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:55.129149 9928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:55.129309 9928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:46:55.139400 9928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:34101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=39423
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:55.140776 9928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:55.142150 9928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:55.155974 9933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:55.156067 9934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:55.156100 9936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:55.156909 9928 server_base.cc:1034] running on GCE node
I20260928 16:46:55.157605 9928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:55.158625 9928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:55.159839 9928 hybrid_clock.cc:648] HybridClock initialized: now 1790614015159788 us; error 41 us; skew 500 ppm
I20260928 16:46:55.160156 9928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:55 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614015, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:55.163425 9928 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:46:55.165033 9928 webserver.cc:460] Webserver started at http://127.26.101.2:39423/ using document root <none> and password file <none>
I20260928 16:46:55.165756 9928 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:55.165884 9928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:55.171749 9928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 16:46:55.174836 9943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.176519 9928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:46:55.176717 9928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "4a760c3939744e3f906f1b6245a65672"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "e7fcb73c8607b90a152771cef31ee45d"
server_key_iv: "8c35bbcde8b3f0b8b5466c00220cdaf5"
server_key_version: "encryptionkey@0"
I20260928 16:46:55.177778 9928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:55.208134 9928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:55.212988 9928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:55.213284 9928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:55.214326 9928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:55.215804 9928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:55.215924 9928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.216042 9928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:55.216114 9928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.248339 9928 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:34101
I20260928 16:46:55.248407 10056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:34101 every 8 connection(s)
I20260928 16:46:55.249940 9928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:46:55.251461 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 9928
I20260928 16:46:55.252192 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:39827
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:45955
--webserver_port=39301
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:46:55.255888 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.253330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:34959)
0928 16:46:55.254285 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.254612 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":354,"threads_started":1}
W20260928 16:46:55.260800 10057 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:45955 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:45955: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
I20260928 16:46:55.261938 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.261524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:58053)
0928 16:46:55.261712 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.261798 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 16:46:55.262974 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.262575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:56119)
0928 16:46:55.262728 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.262829 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 16:46:55.263310 10057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:46:55.591281 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:46:55.591601 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:46:55.591707 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:46:55.599254 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:46:55.599385 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:46:55.599500 10060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:46:55.610311 10060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:39827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=39301
--tserver_master_addrs=127.26.101.62:45955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:46:55.611915 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:46:55.613366 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:46:55.626917 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:55.626991 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:46:55.627389 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:46:55.627840 10060 server_base.cc:1034] running on GCE node
I20260928 16:46:55.628544 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:46:55.629506 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:46:55.630721 10060 hybrid_clock.cc:648] HybridClock initialized: now 1790614015630666 us; error 51 us; skew 500 ppm
I20260928 16:46:55.631155 10060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:46:55 dist-test-slave-t0tw krb5kdc[8599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614015, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:46:55.634459 10060 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:46:55.636032 10060 webserver.cc:460] Webserver started at http://127.26.101.3:39301/ using document root <none> and password file <none>
I20260928 16:46:55.636747 10060 fs_manager.cc:359] Metadata directory not provided
I20260928 16:46:55.636873 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:46:55.642588 10060 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:46:55.645488 10075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.646952 10060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:46:55.647219 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "b6bf48241ac24dabb59f53e88744bb09"
format_stamp: "Formatted at 2026-09-28 16:46:49 on dist-test-slave-t0tw"
server_key: "93749f0f595a7384b88425a9589d45f4"
server_key_iv: "f53ee6f05c6c314d625327aabe9cbfcd"
server_key_version: "encryptionkey@0"
I20260928 16:46:55.648241 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:46:55.673035 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:46:55.677659 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:46:55.677946 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:46:55.679069 10060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:46:55.680589 10060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:46:55.680704 10060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.680815 10060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:46:55.680887 10060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:46:55.713223 10060 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:39827
I20260928 16:46:55.713276 10188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:39827 every 8 connection(s)
I20260928 16:46:55.714706 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:46:55.720145 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.717812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:35877)
0928 16:46:55.718617 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.718889 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":295,"threads_started":1}
I20260928 16:46:55.720206 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10060
W20260928 16:46:55.720963 10189 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:45955 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:45955: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
I20260928 16:46:55.721971 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.721593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:39119)
0928 16:46:55.721779 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.721856 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 16:46:55.722893 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.722598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:59775)
0928 16:46:55.722711 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.722785 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 16:46:55.723176 10189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:46:55.783792 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:55.782727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:38253)
0928 16:46:55.783338 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:55.783467 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":278,"threads_started":1}
I20260928 16:46:56.264834 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:56.263980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:54279)
0928 16:46:56.264476 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:56.264588 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
I20260928 16:46:56.724701 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:56.723750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:57519)
0928 16:46:56.724347 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:56.724462 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":320,"threads_started":1}
I20260928 16:46:56.785902 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:56.784924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:59757)
0928 16:46:56.785579 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:56.785690 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":302,"threads_started":1}
I20260928 16:46:57.266788 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:57.265947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:53715)
0928 16:46:57.266510 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:57.266607 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":288,"threads_started":1}
I20260928 16:46:57.726460 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:57.725704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:41891)
0928 16:46:57.726206 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:57.726292 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260928 16:46:57.787647 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:57.786844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:49891)
0928 16:46:57.787368 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:57.787455 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":306,"threads_started":1}
I20260928 16:46:58.268759 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:58.267870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:51821)
0928 16:46:58.268416 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:58.268518 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":312,"threads_started":1}
I20260928 16:46:58.728294 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:58.727473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:50739)
0928 16:46:58.727966 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:58.728061 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260928 16:46:58.789649 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:58.788740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:58969)
0928 16:46:58.789285 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:58.789399 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":246,"threads_started":1}
I20260928 16:46:59.270701 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:59.269879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:35045)
0928 16:46:59.270427 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:59.270519 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":268,"threads_started":1}
I20260928 16:46:59.730077 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:59.729311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:39151)
0928 16:46:59.729802 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:59.729896 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260928 16:46:59.791608 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:46:59.790731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:33159)
0928 16:46:59.791302 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:46:59.791410 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260928 16:47:00.272702 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.271799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:53903)
0928 16:47:00.272407 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:00.272514 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":344,"threads_started":1}
I20260928 16:47:00.732066 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.731196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:54005)
0928 16:47:00.731724 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:00.731826 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":255,"threads_started":1}
I20260928 16:47:00.793537 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:00.792682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:53993)
0928 16:47:00.793246 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:00.793350 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
I20260928 16:47:01.274732 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.273858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:50687)
0928 16:47:01.274429 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.274542 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260928 16:47:01.733839 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.733086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:32813)
0928 16:47:01.733571 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.733670 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260928 16:47:01.795663 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:01.794702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:59081)
0928 16:47:01.795324 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:01.795421 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":283,"threads_started":1}
I20260928 16:47:02.276609 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.275793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:52589)
0928 16:47:02.276303 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.276392 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":219,"threads_started":1}
I20260928 16:47:02.735769 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.734880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:49211)
0928 16:47:02.735440 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.735540 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":250,"threads_started":1}
I20260928 16:47:02.797415 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:02.796615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:33751)
0928 16:47:02.797143 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:02.797242 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
I20260928 16:47:03.278424 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.277673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:51217)
0928 16:47:03.278158 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.278253 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":222,"threads_started":1}
I20260928 16:47:03.737682 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.736860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:35149)
0928 16:47:03.737420 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.737504 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":270,"threads_started":1}
I20260928 16:47:03.799436 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:03.798535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:60163)
0928 16:47:03.799140 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:03.799241 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":248,"threads_started":1}
I20260928 16:47:04.280426 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:04.279564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:57639)
0928 16:47:04.280120 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:04.280238 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":266,"threads_started":1}
I20260928 16:47:04.739544 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:04.738630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:58581)
0928 16:47:04.739263 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:04.739355 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":326,"threads_started":1}
I20260928 16:47:04.801435 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:04.800573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:48109)
0928 16:47:04.801158 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:04.801260 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
I20260928 16:47:05.282465 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.281584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:56185)
0928 16:47:05.282170 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.282269 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":324,"threads_started":1}
I20260928 16:47:05.741503 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.740573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:43147)
0928 16:47:05.741165 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.741278 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260928 16:47:05.803400 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:05.802538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:39547)
0928 16:47:05.803054 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:05.803193 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260928 16:47:06.284332 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.283502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:42379)
0928 16:47:06.284014 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.284123 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260928 16:47:06.743278 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.742468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:48663)
0928 16:47:06.742943 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.743050 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260928 16:47:06.805330 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:06.804444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:36625)
0928 16:47:06.805042 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:06.805146 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260928 16:47:07.286033 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:07.285237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:49965)
0928 16:47:07.285761 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:07.285847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":310,"threads_started":1}
I20260928 16:47:07.745110 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:07.744316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:35553)
0928 16:47:07.744840 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:07.744943 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":278,"threads_started":1}
I20260928 16:47:07.807524 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:07.806521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:40821)
0928 16:47:07.807115 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:07.807263 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":292,"threads_started":1}
I20260928 16:47:08.287928 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:08.287020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:39245)
0928 16:47:08.287602 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:08.287714 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":244,"threads_started":1}
I20260928 16:47:08.747061 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:08.746200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:56065)
0928 16:47:08.746746 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:08.746850 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260928 16:47:08.809343 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:08.808576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:53159)
0928 16:47:08.809048 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:08.809155 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260928 16:47:09.289691 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:09.288887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:55383)
0928 16:47:09.289398 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:09.289501 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260928 16:47:09.748747 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:09.748001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:34043)
0928 16:47:09.748458 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:09.748562 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260928 16:47:09.811496 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:09.810639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.1:50837)
0928 16:47:09.811187 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:09.811294 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260928 16:47:10.291854 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.291043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.2:42007)
0928 16:47:10.291556 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:10.291664 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":245,"threads_started":1}
I20260928 16:47:10.732272 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9762
I20260928 16:47:10.750754 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.749953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45955 (local address 127.26.101.3:45631)
0928 16:47:10.750455 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:10.750571 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260928 16:47:10.756388 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.733392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.26.101.1:46871)
0928 16:47:10.733996 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.734020 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.734070 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:47:10.736166 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.736176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.736629 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.736973 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:10.738519 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.739851 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.741113 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.741794 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.741981 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.749153 (+ 7172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:10.749169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:10.749193 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:10.749381 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:10.752698 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.753545 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.753552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.753560 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.753776 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.754220 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.754223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.754225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.754833 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:10.755170 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:10.755642 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 16:47:10.756088 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260928 16:47:10.758015 9920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:10.861490 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9762
I20260928 16:47:10.876837 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 9928
I20260928 16:47:10.894474 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.877776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.26.101.2:34101)
0928 16:47:10.878324 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.878338 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.878375 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:47:10.879994 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.880006 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.880437 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.880740 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:10.882168 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.883563 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.884764 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:10.885432 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:10.885570 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:10.888195 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:10.888204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:10.888220 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:10.888266 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:10.891322 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.892043 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.892050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.892058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.892236 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:10.892612 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:10.892615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:10.892617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:10.893152 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:10.893399 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:10.893751 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 16:47:10.894168 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
I20260928 16:47:10.896052 10052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:10.979928 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 9928
I20260928 16:47:10.995602 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10060
I20260928 16:47:11.014292 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:10.996634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.26.101.3:39827)
0928 16:47:10.997264 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:10.997278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:10.997321 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:47:10.998956 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:10.998967 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:10.999468 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:10.999776 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:11.001172 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.002488 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.004011 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:11.004736 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:11.004947 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:11.007747 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:11.007756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:11.007773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:11.007821 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:11.010807 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.011661 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.011669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.011676 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.011856 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:11.012277 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:11.012280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:11.012282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:11.012830 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:11.013122 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:11.013510 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 16:47:11.013984 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":323,"threads_started":1}
I20260928 16:47:11.015784 10184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:11.099460 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10060
2026-09-28T16:47:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24781 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:47:11 dist-test-slave-t0tw krb5kdc[10248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:47:11 dist-test-slave-t0tw krb5kdc[10248](info): set up 2 sockets
Sep 28 16:47:11 dist-test-slave-t0tw krb5kdc[10248](info): commencing operation
krb5kdc: starting...
W20260928 16:47:12.127305 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:47:12 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:47:12.378652 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:12.714942 10261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:12.715340 10261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:12.715493 10261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:12.723372 10261 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:47:12.723496 10261 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:47:12.723541 10261 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:47:12.723610 10261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:12.723683 10261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:12.723794 10261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:12.723852 10261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:12.723928 10261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:12.734138 10261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:12.735507 10261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:12.736878 10261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:12.748999 10266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.749027 10269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:12.748967 10267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:12.749292 10261 server_base.cc:1034] running on GCE node
I20260928 16:47:12.750344 10261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:12.751991 10261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:12.753235 10261 hybrid_clock.cc:648] HybridClock initialized: now 1790614032753165 us; error 44 us; skew 500 ppm
I20260928 16:47:12.753590 10261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:12 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:12.756958 10261 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:47:12.758567 10261 webserver.cc:460] Webserver started at http://127.26.101.62:41869/ using document root <none> and password file <none>
I20260928 16:47:12.759420 10261 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:12.759526 10261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:12.759847 10261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:12.763003 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "8d3cb905e3274a5abac024a4cb4fa7b4"
format_stamp: "Formatted at 2026-09-28 16:47:12 on dist-test-slave-t0tw"
server_key: "d1bbca78f9859faa7cba1a632fde04da"
server_key_iv: "cba3ecb796f37be422f16c9b73d08e50"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.763924 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "8d3cb905e3274a5abac024a4cb4fa7b4"
format_stamp: "Formatted at 2026-09-28 16:47:12 on dist-test-slave-t0tw"
server_key: "d1bbca78f9859faa7cba1a632fde04da"
server_key_iv: "cba3ecb796f37be422f16c9b73d08e50"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.769690 10261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:47:12.773617 10276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:12.775043 10261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 16:47:12.775275 10261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8d3cb905e3274a5abac024a4cb4fa7b4"
format_stamp: "Formatted at 2026-09-28 16:47:12 on dist-test-slave-t0tw"
server_key: "d1bbca78f9859faa7cba1a632fde04da"
server_key_iv: "cba3ecb796f37be422f16c9b73d08e50"
server_key_version: "encryptionkey@0"
I20260928 16:47:12.775507 10261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:12.792955 10261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:12.797366 10261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:12.797636 10261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:12.815953 10261 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:39107
I20260928 16:47:12.815977 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:39107 every 8 connection(s)
I20260928 16:47:12.817327 10261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:12.821126 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:12.823218 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10261
I20260928 16:47:12.823412 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:47:12.824100 27028 external_mini_cluster.cc:1428] Setting key fb91e052d3afb5805690304905f42ef0
I20260928 16:47:12.833170 10328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap starting.
I20260928 16:47:12.837073 10328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:12.838440 10328 log.cc:824] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:47:12 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:12.841979 10328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: No bootstrap required, opened a new log
I20260928 16:47:12.849740 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:12.826109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.26.101.62:39107)
0928 16:47:12.826894 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:12.826906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:12.826947 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:47:12.828670 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:12.828695 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:12.829101 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:12.829463 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:12.831364 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.832712 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.834660 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:12.835472 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:12.835617 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:12.840747 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:12.840755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:12.840770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:12.840812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:12.844102 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.845016 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.845029 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.845039 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.845284 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:12.845805 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:12.845811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:12.845814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:12.846681 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:12.846991 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:12.847797 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 16:47:12.848297 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":362,"threads_started":1}
I20260928 16:47:12.851975 10328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:12.852335 10328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:12.852445 10328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Initialized, Role: FOLLOWER
I20260928 16:47:12.853073 10328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:12.853400 10328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:12.853597 10328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:12.853865 10328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:12.856424 10328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:12.856942 10328 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3cb905e3274a5abac024a4cb4fa7b4; no voters:
I20260928 16:47:12.857525 10328 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:47:12.857650 10334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:12.858316 10334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Running, Role: LEADER
I20260928 16:47:12.858913 10334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:12.859604 10328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:12.861733 10335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:12.862149 10335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:12.864233 10340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:12.864099 10336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3cb905e3274a5abac024a4cb4fa7b4. Latest consensus state: current_term: 1 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:12.864404 10336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:12.869792 10340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:12.880062 10340 catalog_manager.cc:1347] Generated new cluster ID: beb388c7027e477ab68dde1a172b0849
I20260928 16:47:12.880183 10340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:12.897799 10340 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:47:12.899883 10340 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:12.911161 10340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Generated new TSK 0
I20260928 16:47:12.912420 10340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:47:12.979683 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:13.326231 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:13.326545 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:13.326674 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:13.334142 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:13.334295 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:13.334460 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:47:13.344650 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:13.345990 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:13.347370 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:13.360554 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.360634 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.360620 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:13.361620 10355 server_base.cc:1034] running on GCE node
I20260928 16:47:13.362303 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:13.363370 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:13.364588 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790614033364538 us; error 46 us; skew 500 ppm
I20260928 16:47:13.364920 10355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:13 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:13.368247 10355 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:47:13.369819 10355 webserver.cc:460] Webserver started at http://127.26.101.1:38903/ using document root <none> and password file <none>
I20260928 16:47:13.370540 10355 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:13.370663 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:13.370965 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:13.373996 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "709069c2b6dacb6b62589309ae985b51"
server_key_iv: "3734e4578c41aa60019e0dc1c58cb2fe"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.374794 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "709069c2b6dacb6b62589309ae985b51"
server_key_iv: "3734e4578c41aa60019e0dc1c58cb2fe"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.380369 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:47:13.384097 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.385619 10355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:13.385818 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "709069c2b6dacb6b62589309ae985b51"
server_key_iv: "3734e4578c41aa60019e0dc1c58cb2fe"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.386023 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:13.407514 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:13.412082 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:13.412377 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:13.413473 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:13.414873 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:13.414969 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.415118 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:13.415181 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 16:47:13.447503 10355 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41347
I20260928 16:47:13.447598 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41347 every 8 connection(s)
I20260928 16:47:13.448837 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:13.457767 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10355
I20260928 16:47:13.457960 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:47:13.458606 27028 external_mini_cluster.cc:1428] Setting key 5aba43e89cf0e1414872b92384b2717b
Sep 28 16:47:13 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:13.474190 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.451741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:47917 (local address 127.26.101.62:39107)
0928 16:47:13.452277 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:13.452282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:13.453381 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 16:47:13.455417 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:13.455423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:13.455513 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:13.455618 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:13.457520 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.458371 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.460843 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.461383 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.461503 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:13.466188 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:13.466195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:13.466199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:13.466245 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:13.469979 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:13.471224 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:13.471231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:13.471235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:13.471421 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:13.472034 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:13.472039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:13.472042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:13.472403 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:13.472556 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:13.473471 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 16:47:13.473725 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":305,"threads_started":1}
I20260928 16:47:13.475124 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.452331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:47917)
0928 16:47:13.453180 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.453233 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:13.455053 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:13.455930 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:13.455940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:13.456551 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:13.457207 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:13.457229 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.458570 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.458577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:13.460654 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:13.460673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.461663 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.461672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:13.462022 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:13.462826 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:13.462860 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:13.465855 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:13.470209 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:13.470220 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:13.470237 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:13.471006 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:13.471591 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:13.471595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:13.471598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:13.471882 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:13.472578 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:13.472585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:13.473023 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 16:47:13.473306 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 16:47:13.473785 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":299,"threads_started":1}
I20260928 16:47:13.476878 10484 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:13.477272 10484 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:13.478224 10484 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:13.481087 10293 ts_manager.cc:194] Registered new tserver with Master: dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347)
I20260928 16:47:13.482687 10293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:47917
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:47:13.519420 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:13.866313 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:13.866612 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:13.866734 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:13.874300 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:13.874428 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:13.874581 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:47:13.884974 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:13.886381 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:13.887856 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:13.902426 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.902465 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:13.902637 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:13.903270 10490 server_base.cc:1034] running on GCE node
I20260928 16:47:13.903987 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:13.905068 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:13.906306 10490 hybrid_clock.cc:648] HybridClock initialized: now 1790614033906246 us; error 45 us; skew 500 ppm
I20260928 16:47:13.906682 10490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:13 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:13.910403 10490 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:47:13.912199 10490 webserver.cc:460] Webserver started at http://127.26.101.2:40607/ using document root <none> and password file <none>
I20260928 16:47:13.913064 10490 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:13.913210 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:13.913545 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:13.917076 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "2ac666b95456471fa48ee13e7396df79"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "32d633e2aa417576a785b83846b97c0c"
server_key_iv: "fe0e9949ec3d75c85228e8549ee2758d"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.917934 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "2ac666b95456471fa48ee13e7396df79"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "32d633e2aa417576a785b83846b97c0c"
server_key_iv: "fe0e9949ec3d75c85228e8549ee2758d"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.923903 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:47:13.928017 10505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.929546 10490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:47:13.929734 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "2ac666b95456471fa48ee13e7396df79"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "32d633e2aa417576a785b83846b97c0c"
server_key_iv: "fe0e9949ec3d75c85228e8549ee2758d"
server_key_version: "encryptionkey@0"
I20260928 16:47:13.929953 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:13.943192 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:13.948036 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:13.948342 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:13.949383 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:13.950860 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:13.950941 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:13.951092 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:13.951157 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 16:47:13.985822 10490 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:32857
I20260928 16:47:13.985904 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:32857 every 8 connection(s)
I20260928 16:47:13.987187 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:13.996095 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10490
I20260928 16:47:13.996301 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:47:13.996889 27028 external_mini_cluster.cc:1428] Setting key 18fc19c8806b5f5c8daf92126c935626
Sep 28 16:47:13 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614033, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:14.009682 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.989991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:47413 (local address 127.26.101.62:39107)
0928 16:47:13.990512 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:13.990516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:13.991666 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 16:47:13.993630 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:13.993635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:13.993716 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:13.993826 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:13.995704 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.996833 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.998818 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.999400 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.999527 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.003795 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.003801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.003804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.003842 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.006421 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.007268 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.007272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.007274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.007429 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.007881 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.007885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.007887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.008165 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.008302 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.009028 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.009273 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":247,"threads_started":1}
I20260928 16:47:14.010716 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:13.990609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:47413)
0928 16:47:13.991494 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:13.991541 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:13.993311 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:13.994116 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:13.994126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:13.994775 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:13.995463 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:13.995485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.997070 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.997075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:13.998632 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:13.998653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:13.999661 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:13.999670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:13.999926 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.000746 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:14.000772 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:14.003512 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:14.006593 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.006604 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.006621 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.007131 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.007542 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.007548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.007552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.007757 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.008287 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:14.008296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:14.008658 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.008914 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.009494 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260928 16:47:14.012427 10618 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:14.012766 10618 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:14.013515 10618 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:14.015136 10293 ts_manager.cc:194] Registered new tserver with Master: 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857)
I20260928 16:47:14.016306 10293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:47413
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:47:14.051568 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:14.397914 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:14.398223 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:14.398319 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:14.405817 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:14.406044 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:14.406186 10624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:47:14.416412 10624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:14.417709 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:14.419055 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:14.432991 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:14.433004 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:14.433247 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:14.434041 10624 server_base.cc:1034] running on GCE node
I20260928 16:47:14.434777 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:14.435936 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:14.437182 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790614034437129 us; error 51 us; skew 500 ppm
I20260928 16:47:14.437543 10624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:14 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614034, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:14.440987 10624 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:47:14.442625 10624 webserver.cc:460] Webserver started at http://127.26.101.3:41465/ using document root <none> and password file <none>
I20260928 16:47:14.443459 10624 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:14.443588 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:14.443894 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:47:14.447031 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "4df4368dba184ed28df8fc66845fc286"
format_stamp: "Formatted at 2026-09-28 16:47:14 on dist-test-slave-t0tw"
server_key: "6978134a917ae8f57f10837ff95bc275"
server_key_iv: "f0ff5aac81c5ecbc7ba9c9f613e5c3a2"
server_key_version: "encryptionkey@0"
I20260928 16:47:14.447937 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "4df4368dba184ed28df8fc66845fc286"
format_stamp: "Formatted at 2026-09-28 16:47:14 on dist-test-slave-t0tw"
server_key: "6978134a917ae8f57f10837ff95bc275"
server_key_iv: "f0ff5aac81c5ecbc7ba9c9f613e5c3a2"
server_key_version: "encryptionkey@0"
I20260928 16:47:14.453347 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:47:14.457013 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:14.458581 10624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:14.458755 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "4df4368dba184ed28df8fc66845fc286"
format_stamp: "Formatted at 2026-09-28 16:47:14 on dist-test-slave-t0tw"
server_key: "6978134a917ae8f57f10837ff95bc275"
server_key_iv: "f0ff5aac81c5ecbc7ba9c9f613e5c3a2"
server_key_version: "encryptionkey@0"
I20260928 16:47:14.458988 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:14.487174 10484 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
I20260928 16:47:14.493007 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:14.497898 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:14.498179 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:14.499320 10624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:14.500757 10624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:14.500847 10624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:14.500978 10624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:14.501034 10624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:14.532830 10624 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:42523
I20260928 16:47:14.532932 10752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:42523 every 8 connection(s)
I20260928 16:47:14.534133 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:14.540634 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10624
I20260928 16:47:14.540864 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:47:14.541436 27028 external_mini_cluster.cc:1428] Setting key 43523960bb50c2df553aa955d371e85f
Sep 28 16:47:14 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614034, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:14.558408 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.537198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:54087 (local address 127.26.101.62:39107)
0928 16:47:14.537730 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.537734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.538822 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 16:47:14.541087 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.541092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.541172 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.541273 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:14.544063 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.545288 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.547211 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.547627 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.547750 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.551613 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.551618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.551621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.551665 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.554468 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.555470 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.555475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.555477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.555632 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.556269 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.556275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.556277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.556663 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.556827 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.557718 (+ 891us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.557982 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
I20260928 16:47:14.559504 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.537792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:54087)
0928 16:47:14.538612 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.538671 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.540726 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.541599 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.541613 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.542318 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:14.543796 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.543831 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.545499 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.545505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.547024 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.547041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.547899 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.547907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.548153 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.548866 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:14.548902 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:14.551343 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:14.554648 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.554657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.554673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.555251 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.555809 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:14.555813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:14.555816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:14.556092 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:14.556810 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:14.556819 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:14.557279 (+ 460us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.557586 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.558165 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
I20260928 16:47:14.561313 10753 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:14.561614 10753 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:14.562505 10753 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:14.564250 10293 ts_manager.cc:194] Registered new tserver with Master: 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523)
I20260928 16:47:14.565075 10293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:54087
I20260928 16:47:14.571374 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:14.590342 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.576090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.26.101.62:39107)
0928 16:47:14.576304 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.576311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.576697 (+ 386us) server_negotiation.cc:407] Connection header received
0928 16:47:14.576852 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.576856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.576928 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.577014 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:14.578491 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.579350 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.580902 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.581358 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.581478 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.584143 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.584148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.584151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.584190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.587299 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.588174 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.588180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.588182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.588330 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.588749 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.588753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.588756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.589139 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.589361 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.589850 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.590083 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:47:14.594892 10293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:14.602219 10293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:14.632823 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.617971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.26.101.2:32857)
0928 16:47:14.618620 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.618630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.619158 (+ 528us) server_negotiation.cc:407] Connection header received
0928 16:47:14.619362 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.619372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.619685 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.619889 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.621774 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.625032 (+ 3258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.629976 (+ 4944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.631392 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.631757 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.631998 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.632281 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.632476 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":300,"threads_started":1}
I20260928 16:47:14.632903 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.617609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:32857 (local address 127.0.0.1:48424)
0928 16:47:14.618892 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.618909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.619030 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.620208 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.620217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.620253 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.621536 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.621554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.625357 (+ 3803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.625364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.629713 (+ 4349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.629741 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.631661 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.631668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.631993 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.632069 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.632239 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.632455 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":31,"thread_start_us":1018,"threads_started":1}
I20260928 16:47:14.637331 10553 tablet_service.cc:1467] Processing CreateTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 (DEFAULT_TABLE table=test-table [id=fcf940f3716645508a032460d2e60f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:14.638902 10553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcaac34f29404caf9ce2e6fa6b8f1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:14.639309 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.619601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.26.101.3:42523)
0928 16:47:14.628460 (+ 8859us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.628470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.630720 (+ 2250us) server_negotiation.cc:407] Connection header received
0928 16:47:14.630847 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.630857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.631197 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.631423 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.632493 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.634241 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.636377 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.638084 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.638254 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.638497 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.638743 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.638944 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8650,"thread_start_us":8481,"threads_started":1}
I20260928 16:47:14.639716 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.615891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41347 (local address 127.0.0.1:38632)
0928 16:47:14.617089 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.617136 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.617312 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.631438 (+ 14126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.631451 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.631485 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.632060 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.632079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.634018 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.634023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.636377 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.636394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.638427 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.638435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.638614 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.638640 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.638711 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.638879 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":53,"thread_start_us":811,"threads_started":1}
I20260928 16:47:14.639865 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.616211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.26.101.1:41347)
0928 16:47:14.630839 (+ 14628us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.630849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.630873 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:47:14.630965 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.630984 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.631194 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.631441 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.632325 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.633843 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.636616 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.638140 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.639060 (+ 920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.639326 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.639424 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.639590 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14433,"thread_start_us":14256,"threads_started":1}
I20260928 16:47:14.640319 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.618961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42523 (local address 127.0.0.1:48150)
0928 16:47:14.630354 (+ 11393us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.630378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.630543 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.631754 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.631758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.631781 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.632255 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.632269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.634411 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.634416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.636191 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.636207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.638344 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.638352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.638537 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.638574 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.638681 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.638852 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11264,"mutex_wait_us":5,"thread_start_us":11121,"threads_started":1}
I20260928 16:47:14.643404 10418 tablet_service.cc:1467] Processing CreateTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 (DEFAULT_TABLE table=test-table [id=fcf940f3716645508a032460d2e60f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:14.643661 10688 tablet_service.cc:1467] Processing CreateTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 (DEFAULT_TABLE table=test-table [id=fcf940f3716645508a032460d2e60f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:14.644707 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcaac34f29404caf9ce2e6fa6b8f1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:14.645076 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcaac34f29404caf9ce2e6fa6b8f1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:14.655964 10772 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Bootstrap starting.
I20260928 16:47:14.658258 10773 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Bootstrap starting.
I20260928 16:47:14.660009 10774 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Bootstrap starting.
I20260928 16:47:14.661710 10772 tablet_bootstrap.cc:654] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:14.662325 10773 tablet_bootstrap.cc:654] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:14.662953 10774 tablet_bootstrap.cc:654] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:14.663239 10772 log.cc:824] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:14.663662 10773 log.cc:824] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:14.664040 10774 log.cc:824] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:14.666961 10774 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: No bootstrap required, opened a new log
I20260928 16:47:14.667263 10773 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: No bootstrap required, opened a new log
I20260928 16:47:14.667263 10772 tablet_bootstrap.cc:492] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: No bootstrap required, opened a new log
I20260928 16:47:14.667280 10774 ts_tablet_manager.cc:1397] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:47:14.667547 10773 ts_tablet_manager.cc:1397] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 16:47:14.667549 10772 ts_tablet_manager.cc:1397] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 16:47:14.675498 10774 raft_consensus.cc:348] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.675923 10774 raft_consensus.cc:374] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:14.676025 10774 raft_consensus.cc:729] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df4368dba184ed28df8fc66845fc286, State: Initialized, Role: FOLLOWER
I20260928 16:47:14.676693 10774 consensus_queue.cc:260] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.677955 10753 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
I20260928 16:47:14.678736 10774 ts_tablet_manager.cc:1428] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 16:47:14.678642 10773 raft_consensus.cc:348] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.678642 10772 raft_consensus.cc:348] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.679253 10772 raft_consensus.cc:374] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:14.679252 10773 raft_consensus.cc:374] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:14.679404 10773 raft_consensus.cc:729] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac666b95456471fa48ee13e7396df79, State: Initialized, Role: FOLLOWER
I20260928 16:47:14.679476 10772 raft_consensus.cc:729] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb705125d1a4cbe96dbb4a130f5d3ba, State: Initialized, Role: FOLLOWER
I20260928 16:47:14.680262 10773 consensus_queue.cc:260] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.680261 10772 consensus_queue.cc:260] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.683229 10773 ts_tablet_manager.cc:1428] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 16:47:14.683295 10772 ts_tablet_manager.cc:1428] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260928 16:47:14.683876 10618 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
W20260928 16:47:14.701359 10485 tablet.cc:2369] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:47:14.739249 10619 tablet.cc:2369] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:47:14.786159 10754 tablet.cc:2369] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:14.802392 10779 raft_consensus.cc:482] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:14.802712 10779 raft_consensus.cc:504] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.804462 10779 leader_election.cc:288] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347), 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857)
I20260928 16:47:14.810535 10780 raft_consensus.cc:482] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:14.810890 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.804746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41347 (local address 127.26.101.3:50235)
0928 16:47:14.804953 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.804974 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.805104 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.805679 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.805685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.805714 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.805893 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.805904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.807280 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.807286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.808752 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.808764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.810064 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.810070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.810270 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.810306 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.810403 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.810561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260928 16:47:14.810878 10780 raft_consensus.cc:504] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.811512 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.805413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:45223 (local address 127.26.101.2:32857)
0928 16:47:14.805643 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.805648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.805854 (+ 206us) server_negotiation.cc:407] Connection header received
0928 16:47:14.805988 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.805994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.806066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.806132 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.806733 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.807693 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.809535 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.810499 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.810634 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.810705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.811133 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.811271 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:47:14.811702 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.804986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:50235 (local address 127.26.101.1:41347)
0928 16:47:14.805181 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.805187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.805209 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:14.805288 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.805294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.805370 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.805475 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.806057 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.807145 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.808909 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.809918 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.810039 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.810093 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.811316 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.811463 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:47:14.812122 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.805149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:32857 (local address 127.26.101.3:45223)
0928 16:47:14.805753 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.805770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.805874 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.806182 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.806186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.806212 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.806600 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.806612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.807834 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.807838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.809390 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.809405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.810698 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.810704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.810900 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.810931 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.811682 (+ 751us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.811811 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":321,"threads_started":1}
I20260928 16:47:14.812960 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "4df4368dba184ed28df8fc66845fc286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" is_pre_election: true
I20260928 16:47:14.813151 10780 leader_election.cc:288] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857), 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523)
I20260928 16:47:14.813124 10573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "4df4368dba184ed28df8fc66845fc286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac666b95456471fa48ee13e7396df79" is_pre_election: true
I20260928 16:47:14.813434 10438 raft_consensus.cc:2454] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df4368dba184ed28df8fc66845fc286 in term 0.
I20260928 16:47:14.813669 10573 raft_consensus.cc:2454] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df4368dba184ed28df8fc66845fc286 in term 0.
I20260928 16:47:14.814213 10643 leader_election.cc:302] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df4368dba184ed28df8fc66845fc286, dfb705125d1a4cbe96dbb4a130f5d3ba; no voters:
I20260928 16:47:14.815275 10779 raft_consensus.cc:2781] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:14.815440 10779 raft_consensus.cc:482] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:14.815590 10779 raft_consensus.cc:3037] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:14.819108 10779 raft_consensus.cc:504] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.820478 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "4df4368dba184ed28df8fc66845fc286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
I20260928 16:47:14.820717 10438 raft_consensus.cc:3037] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:14.822266 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.813681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:33313 (local address 127.26.101.2:32857)
0928 16:47:14.813901 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.813906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.814470 (+ 564us) server_negotiation.cc:407] Connection header received
0928 16:47:14.814700 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.814705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.814777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.814837 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.815687 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.816795 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.819526 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.820785 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.820924 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.820998 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.821650 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.821983 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 16:47:14.823189 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.813384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:32857 (local address 127.26.101.1:33313)
0928 16:47:14.814376 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.814395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.814544 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.815045 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.815053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.815108 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.815480 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.815496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.816962 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.816967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.819299 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.819317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.821206 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.821212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.821406 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.821441 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.822566 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.822742 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":36,"thread_start_us":702,"threads_started":1}
I20260928 16:47:14.823475 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.814796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:48807 (local address 127.26.101.3:42523)
0928 16:47:14.815177 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.815182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.815542 (+ 360us) server_negotiation.cc:407] Connection header received
0928 16:47:14.815855 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.815864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.815937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.816012 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:14.816756 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.817881 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.820555 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.821983 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.822143 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.822212 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.823100 (+ 888us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.823248 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260928 16:47:14.823930 10779 leader_election.cc:288] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [CANDIDATE]: Term 1 election: Requested vote from peers dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347), 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857)
I20260928 16:47:14.823915 10573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac666b95456471fa48ee13e7396df79" is_pre_election: true
I20260928 16:47:14.824193 10573 raft_consensus.cc:2454] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb705125d1a4cbe96dbb4a130f5d3ba in term 0.
I20260928 16:47:14.824232 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.814519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42523 (local address 127.26.101.1:48807)
0928 16:47:14.815377 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:14.815476 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:14.815702 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:14.816203 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:14.816208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:14.816231 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:14.816574 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.816585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.818083 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.818088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.820359 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:14.820377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.822672 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.822682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:14.822860 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.822909 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:47:14.823696 (+ 787us) client_negotiation.cc:240] Negotiation successful
0928 16:47:14.823844 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":1,"thread_start_us":486,"threads_started":1}
I20260928 16:47:14.824407 10438 raft_consensus.cc:2454] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df4368dba184ed28df8fc66845fc286 in term 1.
I20260928 16:47:14.824892 10573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "4df4368dba184ed28df8fc66845fc286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac666b95456471fa48ee13e7396df79"
I20260928 16:47:14.824910 10373 leader_election.cc:302] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac666b95456471fa48ee13e7396df79, dfb705125d1a4cbe96dbb4a130f5d3ba; no voters:
I20260928 16:47:14.825022 10643 leader_election.cc:302] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df4368dba184ed28df8fc66845fc286, dfb705125d1a4cbe96dbb4a130f5d3ba; no voters:
I20260928 16:47:14.825117 10573 raft_consensus.cc:3037] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:14.825469 10780 raft_consensus.cc:2741] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 16:47:14.825852 10779 raft_consensus.cc:2781] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:14.826445 10708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcaac34f29404caf9ce2e6fa6b8f1055" candidate_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df4368dba184ed28df8fc66845fc286" is_pre_election: true
I20260928 16:47:14.827515 10779 raft_consensus.cc:686] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 1 LEADER]: Becoming Leader. State: Replica: 4df4368dba184ed28df8fc66845fc286, State: Running, Role: LEADER
I20260928 16:47:14.828217 10573 raft_consensus.cc:2454] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df4368dba184ed28df8fc66845fc286 in term 1.
I20260928 16:47:14.828135 10779 consensus_queue.cc:237] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } }
I20260928 16:47:14.834340 10292 catalog_manager.cc:5495] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df4368dba184ed28df8fc66845fc286 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "4df4368dba184ed28df8fc66845fc286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } health_report { overall_health: HEALTHY } } }
I20260928 16:47:14.872289 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.866491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.26.101.3:42523)
0928 16:47:14.866656 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.866661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.866679 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:47:14.866770 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.866774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.866881 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.867007 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:14.867543 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.868391 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.869910 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.870342 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.870456 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.871007 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:14.871612 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:14.871762 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.871862 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.871954 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:47:14.881234 10438 raft_consensus.cc:1261] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Refusing update from remote peer 4df4368dba184ed28df8fc66845fc286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:14.881274 10573 raft_consensus.cc:1261] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Refusing update from remote peer 4df4368dba184ed28df8fc66845fc286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:14.882143 10785 consensus_queue.cc:1035] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:14.882506 10779 consensus_queue.cc:1035] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:14.893718 10789 mvcc.cc:205] Tried to move back new op lower bound from 7334355086862946304 to 7334355086661160960. Current Snapshot: MvccSnapshot[applied={T|T < 7334355086862946304}]
I20260928 16:47:14.894016 10788 mvcc.cc:205] Tried to move back new op lower bound from 7334355086862946304 to 7334355086661160960. Current Snapshot: MvccSnapshot[applied={T|T < 7334355086862946304}]
I20260928 16:47:14.909484 10292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:14.909781 10292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:14.913787 10292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Sending DeleteTablet for 3 replicas of tablet bcaac34f29404caf9ce2e6fa6b8f1055
I20260928 16:47:14.915339 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:14 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:38632
I20260928 16:47:14.915544 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10355
I20260928 16:47:14.915578 10553 tablet_service.cc:1514] Processing DeleteTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:14 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:48424
I20260928 16:47:14.916584 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet bcaac34f29404caf9ce2e6fa6b8f1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:14 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:48150
I20260928 16:47:14.926244 10799 tablet_replica.cc:330] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: stopping tablet replica
I20260928 16:47:14.926833 10799 raft_consensus.cc:2229] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:14.927490 10799 raft_consensus.cc:2258] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:14.930398 10799 ts_tablet_manager.cc:1905] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:14.933847 10798 tablet_replica.cc:330] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: stopping tablet replica
I20260928 16:47:14.934393 10801 tablet_replica.cc:330] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: stopping tablet replica
I20260928 16:47:14.934607 10798 raft_consensus.cc:2229] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:14.935329 10798 raft_consensus.cc:2258] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:14.937930 10798 ts_tablet_manager.cc:1905] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:14.938560 10799 ts_tablet_manager.cc:1918] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:14.938748 10799 log.cc:1196] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/bcaac34f29404caf9ce2e6fa6b8f1055
I20260928 16:47:14.934816 10801 raft_consensus.cc:2229] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:14.939685 10799 ts_tablet_manager.cc:1939] T bcaac34f29404caf9ce2e6fa6b8f1055 P 2ac666b95456471fa48ee13e7396df79: Deleting consensus metadata
I20260928 16:47:14.939903 10801 raft_consensus.cc:2258] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:14.941656 10279 catalog_manager.cc:4841] TS 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857): tablet bcaac34f29404caf9ce2e6fa6b8f1055 (table test-table [id=fcf940f3716645508a032460d2e60f58]) successfully deleted
Sep 28 16:47:14 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:47:14.944180 10801 ts_tablet_manager.cc:1905] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:14.945879 10798 ts_tablet_manager.cc:1918] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:14.946143 10798 log.cc:1196] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/bcaac34f29404caf9ce2e6fa6b8f1055
I20260928 16:47:14.946749 10798 ts_tablet_manager.cc:1939] T bcaac34f29404caf9ce2e6fa6b8f1055 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting consensus metadata
I20260928 16:47:14.949482 10279 catalog_manager.cc:4841] TS dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347): tablet bcaac34f29404caf9ce2e6fa6b8f1055 (table test-table [id=fcf940f3716645508a032460d2e60f58]) successfully deleted
I20260928 16:47:14.952029 10801 ts_tablet_manager.cc:1918] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:14.952239 10801 log.cc:1196] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/bcaac34f29404caf9ce2e6fa6b8f1055
I20260928 16:47:14.952811 10801 ts_tablet_manager.cc:1939] T bcaac34f29404caf9ce2e6fa6b8f1055 P 4df4368dba184ed28df8fc66845fc286: Deleting consensus metadata
I20260928 16:47:14.953606 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:14.916557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.26.101.1:41347)
0928 16:47:14.916751 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:14.916757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:14.928577 (+ 11820us) server_negotiation.cc:407] Connection header received
0928 16:47:14.929796 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:14.929803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:14.929931 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:14.930015 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:14.933966 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.935256 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.937433 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:14.938097 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:14.938206 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:14.945303 (+ 7097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:14.945324 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:14.945345 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:14.945414 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:14.949613 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.950666 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.950677 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.950681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.950881 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:14.951449 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:14.951454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:14.951457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:14.951973 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:14.952543 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:14.952731 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:47:14.953227 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:47:14.954342 10478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:14.954664 10279 catalog_manager.cc:4841] TS 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523): tablet bcaac34f29404caf9ce2e6fa6b8f1055 (table test-table [id=fcf940f3716645508a032460d2e60f58]) successfully deleted
I20260928 16:47:15.046908 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10355
I20260928 16:47:15.070170 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10490
Sep 28 16:47:15 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:47:15.087710 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:15.070998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.26.101.2:32857)
0928 16:47:15.071230 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:15.071236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:15.071259 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:47:15.071330 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:15.071335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:15.071419 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:15.071541 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:15.072846 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:15.073984 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:15.075566 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:15.076207 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:15.076341 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:15.081438 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:15.081455 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:15.081471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:15.081511 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:15.084598 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:15.085524 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:15.085533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:15.085535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:15.085682 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:15.086091 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:15.086095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:15.086097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:15.086474 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:15.086721 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:15.087194 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 16:47:15.087412 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 16:47:15.088392 10613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:15.186236 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10490
I20260928 16:47:15.208948 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10624
Sep 28 16:47:15 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:47:15.226856 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:15.209801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.26.101.3:42523)
0928 16:47:15.210057 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:15.210063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:15.210088 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:47:15.210219 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:15.210224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:15.210319 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:15.210411 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:15.211862 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:15.212617 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:15.213939 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:15.214392 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:15.214514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:15.219482 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:15.219507 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:15.219528 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:15.219594 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:15.223722 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:15.224526 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:15.224538 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:15.224541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:15.224723 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:15.225111 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:15.225117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:15.225121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:15.225632 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:15.225991 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:15.226247 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 16:47:15.226492 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 16:47:15.227691 10748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:15.350283 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10624
I20260928 16:47:15.368518 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10261
I20260928 16:47:15.369742 10323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:15.481105 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10261
I20260928 16:47:35.501576 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--webserver_interface=127.26.101.62
--webserver_port=41869
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:35.912894 10806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:35.913288 10806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:35.913424 10806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:35.920974 10806 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:47:35.921101 10806 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:47:35.921170 10806 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:47:35.921259 10806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:35.921334 10806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:35.921433 10806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:35.921496 10806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:35.921576 10806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:35.932116 10806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:35.934022 10806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:35.935621 10806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:35.961392 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:35.961455 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:35.961484 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:35.961861 10806 server_base.cc:1034] running on GCE node
I20260928 16:47:35.963234 10806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:35.965014 10806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:35.966284 10806 hybrid_clock.cc:648] HybridClock initialized: now 1790614055966190 us; error 66 us; skew 500 ppm
I20260928 16:47:35.966830 10806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:35 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614055, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:35.975306 10806 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:47:35.977264 10806 webserver.cc:460] Webserver started at http://127.26.101.62:41869/ using document root <none> and password file <none>
I20260928 16:47:35.978174 10806 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:35.978305 10806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:35.985491 10806 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:47:35.988909 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:35.990579 10806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 16:47:35.990823 10806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8d3cb905e3274a5abac024a4cb4fa7b4"
format_stamp: "Formatted at 2026-09-28 16:47:12 on dist-test-slave-t0tw"
server_key: "d1bbca78f9859faa7cba1a632fde04da"
server_key_iv: "cba3ecb796f37be422f16c9b73d08e50"
server_key_version: "encryptionkey@0"
I20260928 16:47:35.992033 10806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:36.027750 10806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:36.032712 10806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:36.033052 10806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:36.052335 10806 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:39107
I20260928 16:47:36.052376 10873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:39107 every 8 connection(s)
I20260928 16:47:36.054003 10806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:36.055047 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10806
I20260928 16:47:36.055935 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:41347
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=38903
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:36.062126 10874 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:36.066216 10874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap starting.
I20260928 16:47:36.095064 10874 log.cc:824] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:36.111790 10874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:36.112486 10874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap complete.
I20260928 16:47:36.124202 10874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:36.125186 10874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Initialized, Role: FOLLOWER
I20260928 16:47:36.125909 10874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:36.126169 10874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:36.126291 10874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:36.126456 10874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:47:36.130632 10874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:36.131253 10874 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3cb905e3274a5abac024a4cb4fa7b4; no voters:
I20260928 16:47:36.131891 10874 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:47:36.132037 10878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:47:36.133159 10878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 LEADER]: Becoming Leader. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Running, Role: LEADER
I20260928 16:47:36.133863 10878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:36.134258 10874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:36.136454 10880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3cb905e3274a5abac024a4cb4fa7b4. Latest consensus state: current_term: 2 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:36.136868 10880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:36.137418 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:36.137723 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:36.144624 10885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:36.155211 10885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:36.158881 10885 catalog_manager.cc:1259] Loaded cluster ID: beb388c7027e477ab68dde1a172b0849
I20260928 16:47:36.159319 10885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:36.168169 10885 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:36.170050 10885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Loaded TSK: 0
I20260928 16:47:36.216181 10885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Generated new TSK 1
I20260928 16:47:36.216770 10885 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:47:36.455758 10876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:36.456099 10876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:36.456236 10876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:36.463577 10876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:36.463709 10876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:36.463864 10876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:47:36.473815 10876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:41347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=38903
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:36.475155 10876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:36.476493 10876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:36.490907 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:36.490907 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:36.491683 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:36.492025 10876 server_base.cc:1034] running on GCE node
I20260928 16:47:36.492753 10876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:36.493911 10876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:36.495157 10876 hybrid_clock.cc:648] HybridClock initialized: now 1790614056495098 us; error 46 us; skew 500 ppm
I20260928 16:47:36.495525 10876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:36 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614056, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:36.499583 10876 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:47:36.501458 10876 webserver.cc:460] Webserver started at http://127.26.101.1:38903/ using document root <none> and password file <none>
I20260928 16:47:36.502332 10876 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:36.502483 10876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:36.509722 10876 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:47:36.513497 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:36.515347 10876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:36.515527 10876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "709069c2b6dacb6b62589309ae985b51"
server_key_iv: "3734e4578c41aa60019e0dc1c58cb2fe"
server_key_version: "encryptionkey@0"
I20260928 16:47:36.516901 10876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:36.548290 10876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:36.552582 10876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:36.552834 10876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:36.553802 10876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:36.555214 10876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:36.555330 10876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:36.555416 10876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:36.555486 10876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:36.587154 10876 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41347
I20260928 16:47:36.587240 11026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41347 every 8 connection(s)
I20260928 16:47:36.588634 10876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:36.591428 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 10876
I20260928 16:47:36.592408 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:32857
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=40607
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:36 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614056, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:36.614427 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:36.591758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55805 (local address 127.26.101.62:39107)
0928 16:47:36.592569 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:36.592587 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:36.593164 (+ 577us) server_negotiation.cc:407] Connection header received
0928 16:47:36.596049 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:36.596065 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:36.596534 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:36.596968 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:36.598724 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:36.600038 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:36.601634 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:36.602304 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:36.602500 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:36.606529 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:36.606539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:36.606556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:36.606605 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:36.609494 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:36.610521 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:36.610528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:36.610534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:36.610719 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:36.611260 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:36.611264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:36.611266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:36.611822 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:36.611984 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:36.613032 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0928 16:47:36.613381 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":343,"threads_started":1}
I20260928 16:47:36.614725 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:36.591918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:55805)
0928 16:47:36.592940 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:36.592996 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:36.595333 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:36.597220 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:36.597231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:36.597896 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:36.598467 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:36.598487 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:36.600226 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:36.600230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:36.601477 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:36.601492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:36.602553 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:36.602562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:36.602838 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:36.603512 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:36.603537 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:36.606172 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:36.609669 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:36.609689 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:36.609707 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:36.610366 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:36.610837 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:36.610842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:36.610846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:36.611139 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:36.611970 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:36.611979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:36.612438 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 16:47:36.612801 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 16:47:36.613403 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":477,"threads_started":1}
I20260928 16:47:36.617640 11027 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:36.617941 11027 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:36.618675 11027 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:36.620859 10839 ts_manager.cc:194] Registered new tserver with Master: dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347)
I20260928 16:47:36.622501 10839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:55805
W20260928 16:47:36.924324 11031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:36.924643 11031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:36.924786 11031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:36.932133 11031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:36.932276 11031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:36.932425 11031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:47:36.942487 11031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:32857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=40607
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:36.943971 11031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:36.945806 11031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:36.959812 11036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:36.959877 11037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:36.959841 11039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:36.960462 11031 server_base.cc:1034] running on GCE node
I20260928 16:47:36.961138 11031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:36.962155 11031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:36.963354 11031 hybrid_clock.cc:648] HybridClock initialized: now 1790614056963293 us; error 67 us; skew 500 ppm
I20260928 16:47:36.963672 11031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:36 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614056, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:36.966912 11031 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:47:36.968510 11031 webserver.cc:460] Webserver started at http://127.26.101.2:40607/ using document root <none> and password file <none>
I20260928 16:47:36.969254 11031 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:36.969377 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:36.975426 11031 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:47:36.978430 11046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:36.980024 11031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:36.980188 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "2ac666b95456471fa48ee13e7396df79"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "32d633e2aa417576a785b83846b97c0c"
server_key_iv: "fe0e9949ec3d75c85228e8549ee2758d"
server_key_version: "encryptionkey@0"
I20260928 16:47:36.981204 11031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:37.009002 11031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:37.013701 11031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:37.013969 11031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:37.014983 11031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:37.016597 11031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:37.016701 11031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:37.016811 11031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:37.016880 11031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:37.046425 11031 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:32857
I20260928 16:47:37.046514 11159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:32857 every 8 connection(s)
I20260928 16:47:37.048043 11031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:37.048580 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11031
I20260928 16:47:37.049355 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:42523
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=41465
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:47:37 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614056, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:37.071372 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.050867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:40747 (local address 127.26.101.62:39107)
0928 16:47:37.051106 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.051112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.052487 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 16:47:37.054402 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.054408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.054508 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.054596 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.056207 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.056954 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.058687 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.059434 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.059568 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.064028 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:37.064035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:37.064039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:37.064082 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:37.067470 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.068517 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.068522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.068525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.068685 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.069230 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.069235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.069239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.069632 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:37.069796 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.070706 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.070985 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:47:37.072289 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.051384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:40747)
0928 16:47:37.052309 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.052368 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.054125 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.054860 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.054869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.055456 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:37.056033 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.056051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.057110 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.057114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.058501 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.058518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.059677 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.059684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.059967 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.060745 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:37.060782 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:37.063721 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:37.067663 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:37.067673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:37.067692 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:37.068355 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:37.068812 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:37.068817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:37.068822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:37.069089 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:37.069774 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:37.069783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:37.070240 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.070547 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.071117 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":435,"threads_started":1}
I20260928 16:47:37.073915 11160 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:37.074292 11160 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:37.075302 11160 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:37.077111 10839 ts_manager.cc:194] Registered new tserver with Master: 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857)
I20260928 16:47:37.077956 10839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:40747
W20260928 16:47:37.401292 11163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:37.401598 11163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:37.401721 11163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:37.409246 11163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:37.409389 11163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:37.409534 11163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:47:37.419816 11163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=41465
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:37.421169 11163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:37.422564 11163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:37.436033 11169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:37.436002 11172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:37.435968 11170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:37.436726 11163 server_base.cc:1034] running on GCE node
I20260928 16:47:37.437359 11163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:37.438385 11163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:37.439582 11163 hybrid_clock.cc:648] HybridClock initialized: now 1790614057439537 us; error 37 us; skew 500 ppm
I20260928 16:47:37.439903 11163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:37 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614057, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:37.443342 11163 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:47:37.444895 11163 webserver.cc:460] Webserver started at http://127.26.101.3:41465/ using document root <none> and password file <none>
I20260928 16:47:37.445618 11163 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:37.445766 11163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:37.451611 11163 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:47:37.454773 11179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:37.456270 11163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:47:37.456461 11163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "4df4368dba184ed28df8fc66845fc286"
format_stamp: "Formatted at 2026-09-28 16:47:14 on dist-test-slave-t0tw"
server_key: "6978134a917ae8f57f10837ff95bc275"
server_key_iv: "f0ff5aac81c5ecbc7ba9c9f613e5c3a2"
server_key_version: "encryptionkey@0"
I20260928 16:47:37.457520 11163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:37.482736 11163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:37.487447 11163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:37.487715 11163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:37.488747 11163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:37.490198 11163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:37.490309 11163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:37.490414 11163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:37.490480 11163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:37.521931 11163 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:42523
I20260928 16:47:37.522017 11291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:42523 every 8 connection(s)
I20260928 16:47:37.523509 11163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:47:37.529595 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11163
Sep 28 16:47:37 dist-test-slave-t0tw krb5kdc[10248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614057, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:47:37.547472 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.530613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.26.101.62:39107)
0928 16:47:37.531094 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.531101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.531462 (+ 361us) server_negotiation.cc:407] Connection header received
0928 16:47:37.531598 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.531603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.531676 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.531783 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.533822 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.534905 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.536675 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.537266 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.537387 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.540823 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:37.540830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:37.540835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:37.540896 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:37.544417 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.545172 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.545176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.545178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.545325 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.545749 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.545753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.545755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.546138 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:37.546386 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.546769 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.547000 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260928 16:47:37.547681 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.526199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:48817 (local address 127.26.101.62:39107)
0928 16:47:37.526435 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.526443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.527672 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 16:47:37.529612 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.529617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.529707 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.529815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.531747 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.532916 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.535063 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.535755 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.535926 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.540042 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:37.540050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:37.540055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:37.540103 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:37.544076 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.545114 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.545120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.545124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.545273 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.545753 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.545759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.545763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.546105 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:37.546248 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.547029 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.547294 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":76}
I20260928 16:47:37.548913 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.526712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:48817)
0928 16:47:37.527506 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.527550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.529322 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.530114 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.530126 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.530791 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:47:37.531554 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.531582 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.533117 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.533121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.534893 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.534908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.535997 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.536003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.536222 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.536977 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:47:37.537009 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:47:37.539745 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:47:37.544323 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:37.544335 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:37.544358 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:37.544965 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:37.545397 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:47:37.545401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:47:37.545403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:47:37.545632 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:47:37.546239 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:47:37.546248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:47:37.546618 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.546909 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.547539 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":332,"threads_started":1}
I20260928 16:47:37.550594 11292 heartbeater.cc:347] Connected to a master server at 127.26.101.62:39107
I20260928 16:47:37.550968 11292 heartbeater.cc:464] Registering TS with master...
I20260928 16:47:37.551808 11292 heartbeater.cc:511] Master 127.26.101.62:39107 requested a full tablet report, sending...
I20260928 16:47:37.553393 10838 ts_manager.cc:194] Registered new tserver with Master: 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523)
I20260928 16:47:37.554334 10838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:48817
I20260928 16:47:37.563179 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:47:37.570182 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.564243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.26.101.62:39107)
0928 16:47:37.564429 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.564434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.564865 (+ 431us) server_negotiation.cc:407] Connection header received
0928 16:47:37.565069 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.565073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.565234 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.565372 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:37.565861 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.566565 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.568331 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.568803 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.568917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.569404 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:37.569628 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 16:47:37.569946 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46364: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:47:37.570324 11295 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46364: token expired
W20260928 16:47:37.570905 10759 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.101.62:39107)
I20260928 16:47:37.584316 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.571477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.26.101.62:39107)
0928 16:47:37.571628 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.571633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.571669 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:47:37.571711 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.571715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.571777 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.571848 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.573313 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.574142 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.575572 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.575983 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.576081 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.578745 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:37.578750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:37.578754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:37.578795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:37.581397 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.582327 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.582332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.582334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.582481 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.582925 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.582928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.582930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.583361 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:37.583564 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.583827 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.584023 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:47:37.588625 10838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:47:37.598299 10838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:47:37.621117 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.612629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.26.101.2:32857)
0928 16:47:37.613614 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.613624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.613655 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:47:37.613755 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.613763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.614039 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.614270 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:37.614870 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.616150 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.618702 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.619772 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.619912 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.620141 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.620553 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.620733 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":686,"threads_started":1}
I20260928 16:47:37.621157 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.612372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:32857 (local address 127.0.0.1:56000)
0928 16:47:37.612953 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.612973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.613099 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.614261 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.614265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.614289 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:37.614675 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.614690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.616316 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.616321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.618392 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.618407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.620026 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.620033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.620262 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.620335 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.620489 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.620704 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":1,"thread_start_us":280,"threads_started":1}
I20260928 16:47:37.621383 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.611154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.26.101.3:42523)
0928 16:47:37.611936 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.611945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.612060 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:47:37.612307 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.612315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.612590 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.612803 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:37.613901 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.615516 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.618149 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.619821 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.619966 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.620198 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.620847 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.621022 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":429,"threads_started":1}
I20260928 16:47:37.621688 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.610975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42523 (local address 127.0.0.1:42652)
0928 16:47:37.611930 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.611965 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.612112 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.613150 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.613158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.613192 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:37.613589 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.613603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.615675 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.615681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.617982 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.617999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.620317 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.620323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.620503 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.620533 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.620607 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.620728 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":2,"thread_start_us":549,"threads_started":1}
I20260928 16:47:37.624001 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.612996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41347 (local address 127.0.0.1:46964)
0928 16:47:37.613948 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.613965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.614075 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.616221 (+ 2146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.616225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.616253 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:37.616803 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.616816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.618930 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.618936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.620846 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.620867 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.623331 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.623339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.623520 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.623549 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.623618 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.623721 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":1,"thread_start_us":825,"threads_started":1}
I20260928 16:47:37.624774 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.613614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.26.101.1:41347)
0928 16:47:37.615629 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.615638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.615665 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:47:37.615763 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.615771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.616012 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.616212 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:37.617003 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.618746 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.621044 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.623062 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.623231 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.624232 (+ 1001us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.624345 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.624528 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1851,"thread_start_us":1699,"threads_started":1}
I20260928 16:47:37.625947 11027 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
I20260928 16:47:37.626113 11227 tablet_service.cc:1467] Processing CreateTablet for tablet d76b883cd1b34606a7314f80fb43bd28 (DEFAULT_TABLE table=test-table [id=f54e7550f1714145a9460ec9e0228141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:37.626114 11095 tablet_service.cc:1467] Processing CreateTablet for tablet d76b883cd1b34606a7314f80fb43bd28 (DEFAULT_TABLE table=test-table [id=f54e7550f1714145a9460ec9e0228141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:37.627728 11227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76b883cd1b34606a7314f80fb43bd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:37.627728 11095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76b883cd1b34606a7314f80fb43bd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:37.628654 10959 tablet_service.cc:1467] Processing CreateTablet for tablet d76b883cd1b34606a7314f80fb43bd28 (DEFAULT_TABLE table=test-table [id=f54e7550f1714145a9460ec9e0228141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:47:37.630715 10959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d76b883cd1b34606a7314f80fb43bd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:47:37.644629 11310 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Bootstrap starting.
I20260928 16:47:37.647274 11309 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Bootstrap starting.
I20260928 16:47:37.648386 11311 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Bootstrap starting.
I20260928 16:47:37.649735 11310 tablet_bootstrap.cc:654] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:37.651202 11310 log.cc:824] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:37.651753 11309 tablet_bootstrap.cc:654] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:37.652159 11311 tablet_bootstrap.cc:654] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Neither blocks nor log segments found. Creating new log.
I20260928 16:47:37.652992 11309 log.cc:824] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:37.653318 11311 log.cc:824] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:37.654868 11310 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: No bootstrap required, opened a new log
I20260928 16:47:37.655179 11310 ts_tablet_manager.cc:1397] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260928 16:47:37.656200 11309 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: No bootstrap required, opened a new log
I20260928 16:47:37.656265 11311 tablet_bootstrap.cc:492] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: No bootstrap required, opened a new log
I20260928 16:47:37.656507 11309 ts_tablet_manager.cc:1397] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 16:47:37.656507 11311 ts_tablet_manager.cc:1397] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:47:37.663053 11310 raft_consensus.cc:348] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.663491 11310 raft_consensus.cc:374] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:37.663609 11310 raft_consensus.cc:729] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac666b95456471fa48ee13e7396df79, State: Initialized, Role: FOLLOWER
I20260928 16:47:37.664244 11310 consensus_queue.cc:260] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.665359 11160 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
I20260928 16:47:37.666129 11310 ts_tablet_manager.cc:1428] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260928 16:47:37.667165 11309 raft_consensus.cc:348] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.667176 11311 raft_consensus.cc:348] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.667647 11311 raft_consensus.cc:374] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:37.667647 11309 raft_consensus.cc:374] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:47:37.667753 11311 raft_consensus.cc:729] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df4368dba184ed28df8fc66845fc286, State: Initialized, Role: FOLLOWER
I20260928 16:47:37.667764 11309 raft_consensus.cc:729] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb705125d1a4cbe96dbb4a130f5d3ba, State: Initialized, Role: FOLLOWER
I20260928 16:47:37.668404 11309 consensus_queue.cc:260] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.668411 11311 consensus_queue.cc:260] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.671207 11309 ts_tablet_manager.cc:1428] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260928 16:47:37.672358 11292 heartbeater.cc:503] Master 127.26.101.62:39107 was elected leader, sending a full tablet report...
I20260928 16:47:37.673087 11311 ts_tablet_manager.cc:1428] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260928 16:47:37.746826 11315 raft_consensus.cc:482] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:47:37.747193 11315 raft_consensus.cc:504] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.748991 11315 leader_election.cc:288] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523), dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347)
I20260928 16:47:37.755988 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.749444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:54721 (local address 127.26.101.3:42523)
0928 16:47:37.749686 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.749692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.749979 (+ 287us) server_negotiation.cc:407] Connection header received
0928 16:47:37.750188 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.750194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.750274 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.750373 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:37.751058 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.752004 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.754031 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.754941 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.755116 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.755197 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.755585 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.755730 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:47:37.756106 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.749250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:42523 (local address 127.26.101.2:54721)
0928 16:47:37.749871 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.749888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.750043 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.750431 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.750436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.750479 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:37.750911 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.750926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.752152 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.752158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.753862 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.753880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.755220 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.755228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.755397 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.755437 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.755539 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.755659 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":315,"threads_started":1}
I20260928 16:47:37.756232 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.750209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:37291 (local address 127.26.101.1:41347)
0928 16:47:37.750383 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.750388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.750698 (+ 310us) server_negotiation.cc:407] Connection header received
0928 16:47:37.750835 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.750840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.750918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.751015 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:47:37.751618 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.752546 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.754390 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.755246 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.755421 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.755489 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.755869 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.756003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:47:37.756646 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.749943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41347 (local address 127.26.101.2:37291)
0928 16:47:37.750596 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:37.750609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:47:37.750714 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:47:37.751048 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:47:37.751052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:47:37.751098 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:47:37.751434 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.751446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.752685 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.752689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.754217 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:47:37.754233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.755477 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.755482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:47:37.755671 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.755705 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:47:37.755812 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:47:37.755943 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":24,"thread_start_us":441,"threads_started":1}
I20260928 16:47:37.757372 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76b883cd1b34606a7314f80fb43bd28" candidate_uuid: "2ac666b95456471fa48ee13e7396df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df4368dba184ed28df8fc66845fc286" is_pre_election: true
I20260928 16:47:37.757682 10979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76b883cd1b34606a7314f80fb43bd28" candidate_uuid: "2ac666b95456471fa48ee13e7396df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" is_pre_election: true
I20260928 16:47:37.757861 11247 raft_consensus.cc:2454] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac666b95456471fa48ee13e7396df79 in term 0.
I20260928 16:47:37.758324 10979 raft_consensus.cc:2454] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac666b95456471fa48ee13e7396df79 in term 0.
I20260928 16:47:37.758589 11049 leader_election.cc:302] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac666b95456471fa48ee13e7396df79, 4df4368dba184ed28df8fc66845fc286; no voters:
I20260928 16:47:37.758970 11315 raft_consensus.cc:2781] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:47:37.759143 11315 raft_consensus.cc:482] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:47:37.759262 11315 raft_consensus.cc:3037] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:37.761543 11315 raft_consensus.cc:504] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.762173 11315 leader_election.cc:288] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [CANDIDATE]: Term 1 election: Requested vote from peers 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523), dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347)
I20260928 16:47:37.762603 11247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76b883cd1b34606a7314f80fb43bd28" candidate_uuid: "2ac666b95456471fa48ee13e7396df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df4368dba184ed28df8fc66845fc286"
I20260928 16:47:37.762709 10979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d76b883cd1b34606a7314f80fb43bd28" candidate_uuid: "2ac666b95456471fa48ee13e7396df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
I20260928 16:47:37.762821 11247 raft_consensus.cc:3037] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:37.762895 10979 raft_consensus.cc:3037] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:47:37.765136 10979 raft_consensus.cc:2454] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac666b95456471fa48ee13e7396df79 in term 1.
I20260928 16:47:37.765141 11247 raft_consensus.cc:2454] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac666b95456471fa48ee13e7396df79 in term 1.
I20260928 16:47:37.765558 11049 leader_election.cc:302] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac666b95456471fa48ee13e7396df79, 4df4368dba184ed28df8fc66845fc286; no voters:
I20260928 16:47:37.765853 11315 raft_consensus.cc:2781] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:47:37.766291 11315 raft_consensus.cc:686] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac666b95456471fa48ee13e7396df79, State: Running, Role: LEADER
I20260928 16:47:37.766827 11315 consensus_queue.cc:237] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } }
I20260928 16:47:37.771106 10839 catalog_manager.cc:5495] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac666b95456471fa48ee13e7396df79 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "2ac666b95456471fa48ee13e7396df79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac666b95456471fa48ee13e7396df79" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 32857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 } health_report { overall_health: UNKNOWN } } }
W20260928 16:47:37.775209 11293 tablet.cc:2369] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:47:37.800225 11161 tablet.cc:2369] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:37.823537 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.818029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.26.101.2:32857)
0928 16:47:37.818240 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.818245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.818270 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:47:37.818365 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.818368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.818447 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.818596 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:47:37.818940 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.819708 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.821064 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.821459 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.821574 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.822288 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:47:37.822914 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:47:37.823035 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.823172 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.823250 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:47:37.831880 11247 raft_consensus.cc:1261] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Refusing update from remote peer 2ac666b95456471fa48ee13e7396df79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:37.831879 10979 raft_consensus.cc:1261] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Refusing update from remote peer 2ac666b95456471fa48ee13e7396df79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:47:37.832873 11320 consensus_queue.cc:1035] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:47:37.833245 11315 consensus_queue.cc:1035] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df4368dba184ed28df8fc66845fc286" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:47:37.841032 11028 tablet.cc:2369] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:47:37.844867 11323 mvcc.cc:205] Tried to move back new op lower bound from 7334355180869001216 to 7334355180617281536. Current Snapshot: MvccSnapshot[applied={T|T < 7334355180869001216}]
I20260928 16:47:37.847636 11324 mvcc.cc:205] Tried to move back new op lower bound from 7334355180869001216 to 7334355180617281536. Current Snapshot: MvccSnapshot[applied={T|T < 7334355180869001216 or (T in {7334355180869001216})}]
I20260928 16:47:37.850505 11325 mvcc.cc:205] Tried to move back new op lower bound from 7334355180869001216 to 7334355180617281536. Current Snapshot: MvccSnapshot[applied={T|T < 7334355180869001216}]
I20260928 16:47:37.858738 10838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:37.859033 10838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:47:37.862543 10838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Sending DeleteTablet for 3 replicas of tablet d76b883cd1b34606a7314f80fb43bd28
I20260928 16:47:37.864014 10959 tablet_service.cc:1514] Processing DeleteTablet for tablet d76b883cd1b34606a7314f80fb43bd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:37 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:46964
I20260928 16:47:37.864079 11095 tablet_service.cc:1514] Processing DeleteTablet for tablet d76b883cd1b34606a7314f80fb43bd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:37 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:56000
I20260928 16:47:37.864823 11227 tablet_service.cc:1514] Processing DeleteTablet for tablet d76b883cd1b34606a7314f80fb43bd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:47:37 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:42652
I20260928 16:47:37.864934 11333 tablet_replica.cc:330] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: stopping tablet replica
I20260928 16:47:37.865155 11334 tablet_replica.cc:330] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: stopping tablet replica
I20260928 16:47:37.865399 11333 raft_consensus.cc:2229] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:37.865504 11334 raft_consensus.cc:2229] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:47:37.865935 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10876
I20260928 16:47:37.865970 11333 raft_consensus.cc:2258] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:37.866207 11334 raft_consensus.cc:2258] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:37.867985 11335 tablet_replica.cc:330] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: stopping tablet replica
I20260928 16:47:37.868461 11335 raft_consensus.cc:2229] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:47:37.868960 11334 ts_tablet_manager.cc:1905] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:37.869019 11335 raft_consensus.cc:2258] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:47:37.870386 11333 ts_tablet_manager.cc:1905] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:37.873899 11335 ts_tablet_manager.cc:1905] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:47:37.876000 11334 ts_tablet_manager.cc:1918] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:37.876191 11334 log.cc:1196] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/d76b883cd1b34606a7314f80fb43bd28
I20260928 16:47:37.876715 11334 ts_tablet_manager.cc:1939] T d76b883cd1b34606a7314f80fb43bd28 P 2ac666b95456471fa48ee13e7396df79: Deleting consensus metadata
I20260928 16:47:37.878165 11333 ts_tablet_manager.cc:1918] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:37.878598 11333 log.cc:1196] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/d76b883cd1b34606a7314f80fb43bd28
I20260928 16:47:37.879179 11333 ts_tablet_manager.cc:1939] T d76b883cd1b34606a7314f80fb43bd28 P dfb705125d1a4cbe96dbb4a130f5d3ba: Deleting consensus metadata
I20260928 16:47:37.881237 10825 catalog_manager.cc:4841] TS dfb705125d1a4cbe96dbb4a130f5d3ba (127.26.101.1:41347): tablet d76b883cd1b34606a7314f80fb43bd28 (table test-table [id=f54e7550f1714145a9460ec9e0228141]) successfully deleted
I20260928 16:47:37.881459 11335 ts_tablet_manager.cc:1918] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:47:37.881613 11335 log.cc:1196] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/d76b883cd1b34606a7314f80fb43bd28
I20260928 16:47:37.881966 10825 catalog_manager.cc:4841] TS 2ac666b95456471fa48ee13e7396df79 (127.26.101.2:32857): tablet d76b883cd1b34606a7314f80fb43bd28 (table test-table [id=f54e7550f1714145a9460ec9e0228141]) successfully deleted
I20260928 16:47:37.882221 11335 ts_tablet_manager.cc:1939] T d76b883cd1b34606a7314f80fb43bd28 P 4df4368dba184ed28df8fc66845fc286: Deleting consensus metadata
I20260928 16:47:37.883951 10825 catalog_manager.cc:4841] TS 4df4368dba184ed28df8fc66845fc286 (127.26.101.3:42523): tablet d76b883cd1b34606a7314f80fb43bd28 (table test-table [id=f54e7550f1714145a9460ec9e0228141]) successfully deleted
I20260928 16:47:37.884773 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.866731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.26.101.1:41347)
0928 16:47:37.866982 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.866988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.867009 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:47:37.867267 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.867273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.867355 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.867463 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.869228 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.870364 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.872193 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.872890 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.873028 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:37.876728 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:37.876744 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:37.876763 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:37.876817 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:37.881079 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.882316 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.882326 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.882329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.882494 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:37.882978 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:37.882983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:37.882986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:37.883487 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:37.883799 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:37.884274 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 16:47:37.884445 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 16:47:37.885469 11022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:37.977509 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10876
I20260928 16:47:37.994240 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11031
I20260928 16:47:38.008617 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:37.995169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.26.101.2:32857)
0928 16:47:37.995374 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:37.995384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:37.995403 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:47:37.995457 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:37.995461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:37.995544 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:37.995652 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:37.996945 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.997798 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.999182 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:37.999668 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:37.999789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:38.002475 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:38.002494 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:38.002510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:38.002553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:38.005657 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:38.006407 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:38.006414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:38.006417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:38.006569 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:38.006946 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:38.006949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:38.006951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:38.007436 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:38.007783 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:38.008037 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 16:47:38.008252 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:47:38.009279 11155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:38.144156 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11031
I20260928 16:47:38.163607 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11163
I20260928 16:47:38.178814 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:38.164461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.26.101.3:42523)
0928 16:47:38.164758 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 16:47:38.164764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:47:38.164786 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:47:38.164961 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:47:38.164968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:47:38.165148 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:47:38.165291 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:47:38.166895 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:38.167716 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:38.169095 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:47:38.169520 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:47:38.169651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:47:38.172694 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:47:38.172711 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:47:38.172729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:47:38.172769 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:47:38.175867 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:38.176772 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:38.176780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:38.176783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:38.176932 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:47:38.177365 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:47:38.177367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:47:38.177369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:47:38.177742 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:47:38.178005 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 16:47:38.178277 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 16:47:38.178461 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:47:38.179533 11287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:38.269420 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11163
I20260928 16:47:38.284847 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 10806
I20260928 16:47:38.286096 10869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:47:38.406569 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 10806
I20260928 16:47:58.428146 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--webserver_interface=127.26.101.62
--webserver_port=41869
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:47:58.766172 11340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:58.766492 11340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:58.766615 11340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:58.774196 11340 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:47:58.774319 11340 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:47:58.774358 11340 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:47:58.774441 11340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:47:58.774511 11340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:58.774618 11340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:58.774675 11340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:47:58.774749 11340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:47:58.785180 11340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:58.786513 11340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:58.787972 11340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:58.800308 11348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:58.800278 11346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:58.800349 11345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:58.801354 11340 server_base.cc:1034] running on GCE node
I20260928 16:47:58.802249 11340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:58.803865 11340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:58.805107 11340 hybrid_clock.cc:648] HybridClock initialized: now 1790614078805044 us; error 38 us; skew 500 ppm
I20260928 16:47:58.805438 11340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:58 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614078, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:58.813249 11340 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:47:58.815156 11340 webserver.cc:460] Webserver started at http://127.26.101.62:41869/ using document root <none> and password file <none>
I20260928 16:47:58.815953 11340 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:58.816074 11340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:58.821976 11340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:47:58.825115 11355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:58.826613 11340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:58.826789 11340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8d3cb905e3274a5abac024a4cb4fa7b4"
format_stamp: "Formatted at 2026-09-28 16:47:12 on dist-test-slave-t0tw"
server_key: "d1bbca78f9859faa7cba1a632fde04da"
server_key_iv: "cba3ecb796f37be422f16c9b73d08e50"
server_key_version: "encryptionkey@0"
I20260928 16:47:58.827813 11340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:58.846544 11340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:58.851150 11340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:58.851439 11340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:58.869819 11340 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:39107
I20260928 16:47:58.869837 11406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:39107 every 8 connection(s)
I20260928 16:47:58.871433 11340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:47:58.874102 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11340
I20260928 16:47:58.874958 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:41347
--local_ip_for_outbound_sockets=127.26.101.1
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=38903
--webserver_interface=127.26.101.1
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:58.878681 11407 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:47:58.882063 11407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap starting.
I20260928 16:47:58.909976 11407 log.cc:824] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:47:58.931828 11407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:47:58.932554 11407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Bootstrap complete.
I20260928 16:47:58.946633 11407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:58.947973 11407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Initialized, Role: FOLLOWER
I20260928 16:47:58.948712 11407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:58.948961 11407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:47:58.949100 11407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:47:58.949273 11407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:47:58.953415 11407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:58.953927 11407 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3cb905e3274a5abac024a4cb4fa7b4; no voters:
I20260928 16:47:58.954576 11407 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:47:58.954761 11411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:47:58.955715 11411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [term 3 LEADER]: Becoming Leader. State: Replica: 8d3cb905e3274a5abac024a4cb4fa7b4, State: Running, Role: LEADER
I20260928 16:47:58.956401 11411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } }
I20260928 16:47:58.956686 11407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:47:58.959308 11412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:58.959645 11412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:58.958913 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3cb905e3274a5abac024a4cb4fa7b4. Latest consensus state: current_term: 3 leader_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3cb905e3274a5abac024a4cb4fa7b4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 39107 } } }
I20260928 16:47:58.960237 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:47:58.968784 11418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:47:58.977981 11418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:47:58.979684 11418 catalog_manager.cc:1259] Loaded cluster ID: beb388c7027e477ab68dde1a172b0849
I20260928 16:47:58.979878 11418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:47:58.984161 11418 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:47:58.985921 11418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Loaded TSK: 0
I20260928 16:47:58.986186 11418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d3cb905e3274a5abac024a4cb4fa7b4: Loaded TSK: 1
F20260928 16:47:58.987406 11418 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd197bc39ad google::LogMessage::Fail() at ??:0
@ 0x7fd197bc688a google::LogMessage::SendToLog() at ??:0
@ 0x7fd197bc34de google::LogMessage::Flush() at ??:0
@ 0x7fd197bc4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd19b9277d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd19b91673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd1ad5f21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd1ad5f1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd1ad67d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd1ad67d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd1ad6c75e9 std::function<>::operator()() at ??:0
@ 0x7fd1ad5f5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd1ad5f3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd1ad67b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd1ad67b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd19a80f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd19a81291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd19a812799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd19a7edeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd19dac76db start_thread at ??:0
@ 0x7fd196a5d61f clone at ??:0
W20260928 16:47:59.272040 11409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:59.272405 11409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:59.272569 11409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:59.280001 11409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:59.280156 11409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:59.280313 11409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:47:59.290407 11409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:41347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=38903
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:59.291828 11409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:59.293188 11409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:59.306598 11458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:59.306663 11456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:59.306686 11455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:59.307907 11409 server_base.cc:1034] running on GCE node
I20260928 16:47:59.308560 11409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:59.309545 11409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:59.310807 11409 hybrid_clock.cc:648] HybridClock initialized: now 1790614079310737 us; error 58 us; skew 500 ppm
I20260928 16:47:59.311183 11409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:59 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614079, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:59.314404 11409 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:47:59.316027 11409 webserver.cc:460] Webserver started at http://127.26.101.1:38903/ using document root <none> and password file <none>
I20260928 16:47:59.316776 11409 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:59.316908 11409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:59.322430 11409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 16:47:59.325279 11465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.326627 11409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:47:59.326789 11409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "dfb705125d1a4cbe96dbb4a130f5d3ba"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "709069c2b6dacb6b62589309ae985b51"
server_key_iv: "3734e4578c41aa60019e0dc1c58cb2fe"
server_key_version: "encryptionkey@0"
I20260928 16:47:59.327865 11409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:59.340190 11409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:59.344621 11409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:59.344885 11409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:59.346239 11409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:59.347702 11409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:59.347815 11409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.347925 11409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:59.348002 11409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.380823 11409 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41347
I20260928 16:47:59.380884 11577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41347 every 8 connection(s)
I20260928 16:47:59.382314 11409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:47:59.384691 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11409
I20260928 16:47:59.385360 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:32857
--local_ip_for_outbound_sockets=127.26.101.2
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=40607
--webserver_interface=127.26.101.2
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:47:59.387930 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.385700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:53597)
0928 16:47:59.386581 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.386898 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":357,"threads_started":1}
W20260928 16:47:59.388775 11578 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:39107 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:39107: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
I20260928 16:47:59.389746 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.389355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:50281)
0928 16:47:59.389542 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.389634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 16:47:59.390661 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.390343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:45103)
0928 16:47:59.390471 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.390549 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:47:59.390901 11578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:47:59.717535 11581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:47:59.717836 11581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:47:59.717980 11581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:47:59.725440 11581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:47:59.725569 11581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:47:59.725729 11581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:47:59.735900 11581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:32857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=40607
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:47:59.737303 11581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:47:59.738642 11581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:47:59.751515 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:59.751541 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:47:59.751597 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:47:59.752226 11581 server_base.cc:1034] running on GCE node
I20260928 16:47:59.752801 11581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:47:59.753750 11581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:47:59.754979 11581 hybrid_clock.cc:648] HybridClock initialized: now 1790614079754930 us; error 34 us; skew 500 ppm
I20260928 16:47:59.755329 11581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:47:59 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614079, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:47:59.758536 11581 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:47:59.760042 11581 webserver.cc:460] Webserver started at http://127.26.101.2:40607/ using document root <none> and password file <none>
I20260928 16:47:59.760699 11581 fs_manager.cc:359] Metadata directory not provided
I20260928 16:47:59.760799 11581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:47:59.766247 11581 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:47:59.769277 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.770778 11581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:47:59.770956 11581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "2ac666b95456471fa48ee13e7396df79"
format_stamp: "Formatted at 2026-09-28 16:47:13 on dist-test-slave-t0tw"
server_key: "32d633e2aa417576a785b83846b97c0c"
server_key_iv: "fe0e9949ec3d75c85228e8549ee2758d"
server_key_version: "encryptionkey@0"
I20260928 16:47:59.772029 11581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:47:59.789445 11581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:47:59.793803 11581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:47:59.794071 11581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:47:59.795166 11581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:47:59.796552 11581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:47:59.796658 11581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.796763 11581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:47:59.796829 11581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:47:59.829077 11581 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:32857
I20260928 16:47:59.829141 11710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:32857 every 8 connection(s)
I20260928 16:47:59.830646 11581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:47:59.836086 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.833904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:58367)
0928 16:47:59.834774 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.835039 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":432,"threads_started":1}
W20260928 16:47:59.836906 11712 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:39107 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:39107: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
I20260928 16:47:59.837965 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.837524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:48117)
0928 16:47:59.837742 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.837833 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 16:47:59.839152 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:47:59.838690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:44641)
0928 16:47:59.838869 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:47:59.838966 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 16:47:59.839437 11712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:47:59.840015 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11581
I20260928 16:47:59.840629 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:42523
--local_ip_for_outbound_sockets=127.26.101.3
--tserver_master_addrs=127.26.101.62:39107
--webserver_port=41465
--webserver_interface=127.26.101.3
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:00.179706 11715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:00.180030 11715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:00.180159 11715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:00.187988 11715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:00.188133 11715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:00.188295 11715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:00.199324 11715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=41465
--tserver_master_addrs=127.26.101.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:00.200803 11715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:00.202245 11715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:00.216305 11721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:00.216553 11720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:00.216910 11723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:00.217697 11715 server_base.cc:1034] running on GCE node
I20260928 16:48:00.218351 11715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:00.219465 11715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:00.220715 11715 hybrid_clock.cc:648] HybridClock initialized: now 1790614080220653 us; error 45 us; skew 500 ppm
I20260928 16:48:00.221096 11715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:00 dist-test-slave-t0tw krb5kdc[10248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614080, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:00.224534 11715 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:48:00.226199 11715 webserver.cc:460] Webserver started at http://127.26.101.3:41465/ using document root <none> and password file <none>
I20260928 16:48:00.226919 11715 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:00.227047 11715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:00.232962 11715 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:48:00.235963 11730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:00.237516 11715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:00.237701 11715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "4df4368dba184ed28df8fc66845fc286"
format_stamp: "Formatted at 2026-09-28 16:47:14 on dist-test-slave-t0tw"
server_key: "6978134a917ae8f57f10837ff95bc275"
server_key_iv: "f0ff5aac81c5ecbc7ba9c9f613e5c3a2"
server_key_version: "encryptionkey@0"
I20260928 16:48:00.238813 11715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:00.258237 11715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:00.263028 11715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:00.263346 11715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:00.264524 11715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:00.266093 11715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:00.266192 11715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:00.266304 11715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:00.266372 11715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:00.298655 11715 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:42523
I20260928 16:48:00.298733 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:42523 every 8 connection(s)
I20260928 16:48:00.300211 11715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:00.305399 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:00.303497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:53985)
0928 16:48:00.304236 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:00.304473 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260928 16:48:00.305814 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11715
W20260928 16:48:00.306187 11845 heartbeater.cc:644] Failed to heartbeat to 127.26.101.62:39107 (0 consecutive failures): Network error: Failed to ping master at 127.26.101.62:39107: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
I20260928 16:48:00.307299 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:00.306824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:46021)
0928 16:48:00.307016 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:00.307145 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 16:48:00.308194 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:00.307844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:43179)
0928 16:48:00.307996 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:00.308075 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 16:48:00.308461 11845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:48:00.392546 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:00.391668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:36359)
0928 16:48:00.392197 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:00.392287 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":260,"threads_started":1}
I20260928 16:48:00.841100 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:00.840145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:40983)
0928 16:48:00.840702 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:00.840828 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":249,"threads_started":1}
I20260928 16:48:01.309966 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:01.309089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:38603)
0928 16:48:01.309624 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:01.309734 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":233,"threads_started":1}
I20260928 16:48:01.394555 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:01.393702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:49835)
0928 16:48:01.394274 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:01.394369 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":255,"threads_started":1}
I20260928 16:48:01.842820 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:01.842060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:59141)
0928 16:48:01.842533 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:01.842623 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260928 16:48:02.311733 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:02.310920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:41575)
0928 16:48:02.311462 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:02.311551 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
I20260928 16:48:02.396458 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:02.395626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:38273)
0928 16:48:02.396110 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:02.396225 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260928 16:48:02.844635 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:02.843861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:50257)
0928 16:48:02.844333 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:02.844427 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260928 16:48:03.313400 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:03.312618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:36603)
0928 16:48:03.313103 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:03.313202 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260928 16:48:03.398388 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:03.397547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:34063)
0928 16:48:03.398052 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:03.398150 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260928 16:48:03.846366 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:03.845518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:41689)
0928 16:48:03.846059 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:03.846146 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":263,"threads_started":1}
I20260928 16:48:04.315230 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:04.314421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:51345)
0928 16:48:04.314866 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:04.314956 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":213,"threads_started":1}
I20260928 16:48:04.400341 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:04.399517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:36437)
0928 16:48:04.400062 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:04.400162 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260928 16:48:04.848212 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:04.847424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:37011)
0928 16:48:04.847921 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:04.848032 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":241,"threads_started":1}
I20260928 16:48:05.317078 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:05.316267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:41039)
0928 16:48:05.316784 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:05.316896 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":256,"threads_started":1}
I20260928 16:48:05.402165 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:05.401359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:43251)
0928 16:48:05.401877 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:05.401984 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260928 16:48:05.850023 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:05.849179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:50511)
0928 16:48:05.849734 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:05.849844 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":287,"threads_started":1}
I20260928 16:48:06.318975 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:06.318130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:47111)
0928 16:48:06.318684 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:06.318791 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":325,"threads_started":1}
I20260928 16:48:06.404104 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:06.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:48305)
0928 16:48:06.403784 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:06.403871 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":245,"threads_started":1}
I20260928 16:48:06.852087 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:06.851156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:45093)
0928 16:48:06.851720 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:06.851819 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":260,"threads_started":1}
I20260928 16:48:07.320822 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:07.319942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:37915)
0928 16:48:07.320479 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:07.320571 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":235,"threads_started":1}
I20260928 16:48:07.405818 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:07.405041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:40063)
0928 16:48:07.405548 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:07.405648 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":254,"threads_started":1}
I20260928 16:48:07.853843 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:07.853057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:55471)
0928 16:48:07.853553 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:07.853650 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260928 16:48:08.322553 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:08.321765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:34755)
0928 16:48:08.322276 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:08.322370 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260928 16:48:08.407737 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:08.406868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:36701)
0928 16:48:08.407457 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:08.407546 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":278,"threads_started":1}
I20260928 16:48:08.855870 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:08.854871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:52143)
0928 16:48:08.855485 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:08.855609 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":311,"threads_started":1}
I20260928 16:48:09.324287 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:09.323478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:39671)
0928 16:48:09.324011 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:09.324104 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
I20260928 16:48:09.409662 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:09.408838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:37053)
0928 16:48:09.409383 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:09.409481 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
I20260928 16:48:09.857700 11877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:09.856913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:49131)
0928 16:48:09.857410 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:09.857527 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260928 16:48:10.326195 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:10.325388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:49753)
0928 16:48:10.325930 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:10.326025 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":295,"threads_started":1}
I20260928 16:48:10.411511 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:10.410646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:58679)
0928 16:48:10.411213 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:10.411307 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260928 16:48:10.859700 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:10.858785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:43493)
0928 16:48:10.859380 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:10.859506 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":238,"threads_started":1}
I20260928 16:48:11.328094 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:11.327317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:57011)
0928 16:48:11.327804 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:11.327903 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260928 16:48:11.413200 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:11.412477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:51545)
0928 16:48:11.412947 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:11.413041 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260928 16:48:11.861439 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:11.860651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:44161)
0928 16:48:11.861126 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:11.861219 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260928 16:48:12.330106 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.329177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:55817)
0928 16:48:12.329820 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.329932 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":336,"threads_started":1}
I20260928 16:48:12.415062 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.414223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:34991)
0928 16:48:12.414742 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.414881 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":255,"threads_started":1}
I20260928 16:48:12.863261 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:12.862378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:52643)
0928 16:48:12.862916 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:12.863024 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":265,"threads_started":1}
I20260928 16:48:13.331918 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.331016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:46461)
0928 16:48:13.331610 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.331716 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":341,"threads_started":1}
I20260928 16:48:13.416885 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.416083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:39505)
0928 16:48:13.416608 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.416700 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
I20260928 16:48:13.865072 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:13.864309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:43289)
0928 16:48:13.864780 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:13.864880 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260928 16:48:14.333639 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.332876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:58223)
0928 16:48:14.333334 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.333425 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":215,"threads_started":1}
I20260928 16:48:14.418919 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.418106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:36111)
0928 16:48:14.418612 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.418715 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260928 16:48:14.867007 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:14.866226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.2:60877)
0928 16:48:14.866715 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:14.866810 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
I20260928 16:48:15.315869 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11409
I20260928 16:48:15.334467 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.317038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.26.101.1:41347)
0928 16:48:15.317611 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:15.317632 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:15.317858 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:48:15.319682 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:15.319692 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:15.320139 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:15.320471 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:15.322079 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.323357 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.324672 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.325356 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.325532 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:15.328235 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:15.328244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:15.328258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:15.328303 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:15.330932 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.331751 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.331759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.331766 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.331979 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.332394 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.332399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.332403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.332992 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:15.333303 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:15.333735 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 16:48:15.334181 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260928 16:48:15.335659 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.334789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.3:42293)
0928 16:48:15.335294 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:15.335443 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":227,"threads_started":1}
I20260928 16:48:15.336400 11573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:15.421161 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.420147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:39107 (local address 127.26.101.1:59783)
0928 16:48:15.420711 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:15.420897 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":289,"threads_started":1}
I20260928 16:48:15.423580 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11409
I20260928 16:48:15.439716 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11581
I20260928 16:48:15.457706 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.440715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.26.101.2:32857)
0928 16:48:15.441276 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:15.441292 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:15.441329 (+ 37us) server_negotiation.cc:407] Connection header received
0928 16:48:15.442903 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:15.442917 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:15.443478 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:15.443808 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:15.445284 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.446535 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.448056 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.448782 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.448922 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:15.451735 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:15.451743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:15.451759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:15.451808 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:15.454642 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.455345 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.455352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.455359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.455556 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.455875 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.455878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.455880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.456389 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:15.456660 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:15.456995 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 16:48:15.457391 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260928 16:48:15.459275 11706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:15.544147 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11581
I20260928 16:48:15.559827 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11715
I20260928 16:48:15.578151 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:15.560782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.26.101.3:42523)
0928 16:48:15.561363 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:15.561377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:15.561421 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:48:15.563125 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:15.563141 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:15.563598 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:15.563890 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:15.565206 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.566440 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.567964 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:15.568689 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:15.568829 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:15.571622 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:15.571630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:15.571646 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:15.571692 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:15.574616 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.575421 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.575430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.575437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.575637 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:15.576098 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:15.576102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:15.576104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:15.576693 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:15.576960 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:15.577389 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 16:48:15.577839 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":261,"threads_started":1}
I20260928 16:48:15.579649 11840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:15.665441 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11715
2026-09-28T16:48:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64566 ms)
[----------] 30 tests from SecurityITest (305353 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:15 dist-test-slave-t0tw krb5kdc[11903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:15 dist-test-slave-t0tw krb5kdc[11903](info): set up 2 sockets
Sep 28 16:48:15 dist-test-slave-t0tw krb5kdc[11903](info): commencing operation
krb5kdc: starting...
W20260928 16:48:16.695045 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:16 dist-test-slave-t0tw krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:48:16.948426 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37553
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:17.291932 11916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:17.292253 11916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:17.292384 11916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:17.299983 11916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:17.300117 11916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:17.300165 11916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:17.300246 11916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:17.300313 11916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:17.310575 11916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:17.311949 11916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:17.313239 11916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:17.324908 11921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:17.324949 11922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:17.324873 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:17.325662 11916 server_base.cc:1034] running on GCE node
I20260928 16:48:17.326434 11916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:17.327939 11916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:17.329154 11916 hybrid_clock.cc:648] HybridClock initialized: now 1790614097329102 us; error 39 us; skew 500 ppm
I20260928 16:48:17.329528 11916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:17 dist-test-slave-t0tw krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614097, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:17.337311 11916 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:48:17.338922 11916 webserver.cc:460] Webserver started at http://127.26.101.62:42823/ using document root <none> and password file <none>
I20260928 16:48:17.339668 11916 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:17.339793 11916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:17.340085 11916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:17.343240 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "3b5e6893c4c34b06a4580db9e411c521"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "baa94eb9f9956ce890dc7d04de604b0c"
server_key_iv: "5151ca3a1a2d874e0784a3bcd8160f59"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.344025 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "3b5e6893c4c34b06a4580db9e411c521"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "baa94eb9f9956ce890dc7d04de604b0c"
server_key_iv: "5151ca3a1a2d874e0784a3bcd8160f59"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.349486 11916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:17.353065 11932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:17.354437 11916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:17.354609 11916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "3b5e6893c4c34b06a4580db9e411c521"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "baa94eb9f9956ce890dc7d04de604b0c"
server_key_iv: "5151ca3a1a2d874e0784a3bcd8160f59"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.354815 11916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:17.379408 11916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:17.383988 11916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:17.384276 11916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:17.402215 11916 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37553
I20260928 16:48:17.402240 11983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37553 every 8 connection(s)
I20260928 16:48:17.403635 11916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:17.407569 11984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:17.413687 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 11916
I20260928 16:48:17.413908 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:17.414520 27028 external_mini_cluster.cc:1428] Setting key 90836493d3bf46c2baf6572ef44a6126
I20260928 16:48:17.418403 11984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: Bootstrap starting.
I20260928 16:48:17.422075 11984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:17.423175 11984 log.cc:824] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:17.426232 11984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: No bootstrap required, opened a new log
Sep 28 16:48:17 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:17.433734 11984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } }
I20260928 16:48:17.434120 11984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:17.434214 11984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5e6893c4c34b06a4580db9e411c521, State: Initialized, Role: FOLLOWER
I20260928 16:48:17.434965 11984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } }
I20260928 16:48:17.435305 11984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:17.435453 11984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:17.435622 11984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:17.437712 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:17.416675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.26.101.62:37553)
0928 16:48:17.417478 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:17.417489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:17.417531 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:48:17.419348 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:17.419373 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:17.419770 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:17.420099 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:17.421881 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:17.423270 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:17.424891 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:17.425562 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:17.425686 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:17.429866 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:17.429874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:17.429892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:17.429934 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:17.433044 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:17.433686 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:17.433693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:17.433699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:17.433893 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:17.434321 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:17.434327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:17.434329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:17.435052 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:17.435383 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:17.435873 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 16:48:17.436334 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":359,"threads_started":1}
I20260928 16:48:17.438723 11984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } }
I20260928 16:48:17.439347 11984 leader_election.cc:302] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b5e6893c4c34b06a4580db9e411c521; no voters:
I20260928 16:48:17.440016 11984 leader_election.cc:288] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:17.440161 11989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:17.441162 11989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5e6893c4c34b06a4580db9e411c521, State: Running, Role: LEADER
I20260928 16:48:17.441695 11989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } }
I20260928 16:48:17.442016 11984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:17.444190 11991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b5e6893c4c34b06a4580db9e411c521. Latest consensus state: current_term: 1 leader_uuid: "3b5e6893c4c34b06a4580db9e411c521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } } }
I20260928 16:48:17.444968 11991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:17.444996 11990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b5e6893c4c34b06a4580db9e411c521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b5e6893c4c34b06a4580db9e411c521" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37553 } } }
I20260928 16:48:17.445314 11990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:17.447705 11997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:17.452945 11997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:17.462359 11997 catalog_manager.cc:1347] Generated new cluster ID: 59de0736cbd6486babf0314e414f18e2
I20260928 16:48:17.462548 11997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:17.486893 11997 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:17.489075 11997 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:17.497866 11997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: Generated new TSK 0
I20260928 16:48:17.498734 11997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:48:17.558876 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:17.896996 12010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:17.897305 12010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:17.897413 12010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:17.905305 12010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:17.905445 12010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:17.905563 12010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:17.915840 12010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:17.917166 12010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:17.918525 12010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:17.932143 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:17.932199 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:17.932407 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:17.936899 12010 server_base.cc:1034] running on GCE node
I20260928 16:48:17.937646 12010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:17.938772 12010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:17.940030 12010 hybrid_clock.cc:648] HybridClock initialized: now 1790614097939940 us; error 76 us; skew 500 ppm
I20260928 16:48:17.940500 12010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:17 dist-test-slave-t0tw krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614097, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:17.944317 12010 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:48:17.946009 12010 webserver.cc:460] Webserver started at http://127.26.101.1:35585/ using document root <none> and password file <none>
I20260928 16:48:17.946877 12010 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:17.947014 12010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:17.947413 12010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:17.950524 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "7a4aa454a9c74579ab9ff32cb756bf8c"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "274a2f5f57002ad7a901255510783664"
server_key_iv: "3ac1b4ab2f45ec5fa0c3bda499883e2a"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.951409 12010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "7a4aa454a9c74579ab9ff32cb756bf8c"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "274a2f5f57002ad7a901255510783664"
server_key_iv: "3ac1b4ab2f45ec5fa0c3bda499883e2a"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.957384 12010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:48:17.961372 12025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:17.962983 12010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:17.963195 12010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "7a4aa454a9c74579ab9ff32cb756bf8c"
format_stamp: "Formatted at 2026-09-28 16:48:17 on dist-test-slave-t0tw"
server_key: "274a2f5f57002ad7a901255510783664"
server_key_iv: "3ac1b4ab2f45ec5fa0c3bda499883e2a"
server_key_version: "encryptionkey@0"
I20260928 16:48:17.963411 12010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:17.978833 12010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:17.983140 12010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:17.983429 12010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:17.984477 12010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:17.985879 12010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:17.985956 12010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:17.986050 12010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:17.986099 12010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:18.017689 12010 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:39497
I20260928 16:48:18.017735 12140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:39497 every 8 connection(s)
I20260928 16:48:18.019011 12010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:18.024247 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12010
I20260928 16:48:18.024477 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:18.025244 27028 external_mini_cluster.cc:1428] Setting key 0d6005757d2a00fd832b0f7f3a521c4e
Sep 28 16:48:18 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614097, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:18.041477 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.021573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:45385 (local address 127.26.101.62:37553)
0928 16:48:18.022029 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:18.022034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:18.023004 (+ 970us) server_negotiation.cc:407] Connection header received
0928 16:48:18.025391 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:18.025398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:18.025478 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:18.025574 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:18.027441 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.028169 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.030486 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.030940 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.031058 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:18.035339 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:18.035345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:18.035349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:18.035401 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:18.038290 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:18.039114 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:18.039120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:18.039123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:18.039287 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:18.039708 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:18.039713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:18.039716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:18.040051 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:18.040192 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:18.040893 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 16:48:18.041125 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260928 16:48:18.042376 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.022128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37553 (local address 127.26.101.1:45385)
0928 16:48:18.022834 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:18.022873 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:18.025047 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:18.025851 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:18.025863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:18.026542 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:18.027250 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:18.027276 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.028344 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.028350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:18.030267 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:18.030285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.031210 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.031218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:18.031520 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:18.032278 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:18.032311 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:18.035023 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:18.038464 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:18.038471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:18.038487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:18.038958 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:18.039385 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:18.039388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:18.039390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:18.039599 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:18.040178 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:18.040185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:18.040537 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 16:48:18.040763 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 16:48:18.041189 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":265,"threads_started":1}
I20260928 16:48:18.044198 12141 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37553
I20260928 16:48:18.044603 12141 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:18.045385 12141 heartbeater.cc:511] Master 127.26.101.62:37553 requested a full tablet report, sending...
I20260928 16:48:18.047880 11949 ts_manager.cc:194] Registered new tserver with Master: 7a4aa454a9c74579ab9ff32cb756bf8c (127.26.101.1:39497)
I20260928 16:48:18.050026 11949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:45385
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:48:18.086009 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:18.426851 12147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:18.427201 12147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:18.427348 12147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:18.434757 12147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:18.434894 12147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:18.435046 12147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:18.445369 12147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:18.446759 12147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:18.448154 12147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:18.461776 12152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:18.463446 12155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:18.463487 12153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:18.463531 12147 server_base.cc:1034] running on GCE node
I20260928 16:48:18.464252 12147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:18.465301 12147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:18.466552 12147 hybrid_clock.cc:648] HybridClock initialized: now 1790614098466501 us; error 37 us; skew 500 ppm
I20260928 16:48:18.466915 12147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:18 dist-test-slave-t0tw krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614098, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:18.470326 12147 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:48:18.471887 12147 webserver.cc:460] Webserver started at http://127.26.101.2:42843/ using document root <none> and password file <none>
I20260928 16:48:18.472594 12147 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:18.472718 12147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:18.473062 12147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:18.476133 12147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "f08d2d604a2f4c86a850425b70714889"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "c347b7b9f871362fc637978546ee6ca6"
server_key_iv: "e86a10636dfc8b14bc48f96eeed7f7e0"
server_key_version: "encryptionkey@0"
I20260928 16:48:18.476881 12147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "f08d2d604a2f4c86a850425b70714889"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "c347b7b9f871362fc637978546ee6ca6"
server_key_iv: "e86a10636dfc8b14bc48f96eeed7f7e0"
server_key_version: "encryptionkey@0"
I20260928 16:48:18.482259 12147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:18.485903 12163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:18.487318 12147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:18.487490 12147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "f08d2d604a2f4c86a850425b70714889"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "c347b7b9f871362fc637978546ee6ca6"
server_key_iv: "e86a10636dfc8b14bc48f96eeed7f7e0"
server_key_version: "encryptionkey@0"
I20260928 16:48:18.487685 12147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:18.503544 12147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:18.508024 12147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:18.508312 12147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:18.509374 12147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:18.510777 12147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:18.510869 12147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:18.510998 12147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:18.511052 12147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:18.541586 12147 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:33595
I20260928 16:48:18.541649 12276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:33595 every 8 connection(s)
I20260928 16:48:18.543354 12147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:18.551477 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12147
I20260928 16:48:18.551692 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:18.552249 27028 external_mini_cluster.cc:1428] Setting key e96d9d93d25b1c05ec1dbdaf6cc4468c
Sep 28 16:48:18 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614098, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:18.564965 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.545879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:52517 (local address 127.26.101.62:37553)
0928 16:48:18.546325 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:18.546329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:18.547458 (+ 1129us) server_negotiation.cc:407] Connection header received
0928 16:48:18.549335 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:18.549340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:18.549417 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:18.549541 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:18.551117 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.552281 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.554260 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.554851 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.554995 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:18.558625 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:18.558632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:18.558636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:18.558674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:18.561767 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:18.562531 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:18.562537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:18.562540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:18.562693 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:18.563111 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:18.563116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:18.563119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:18.563466 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:18.563612 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:18.564323 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 16:48:18.564576 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":235,"threads_started":1}
I20260928 16:48:18.565963 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:18.546446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37553 (local address 127.26.101.2:52517)
0928 16:48:18.547253 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:18.547301 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:18.548990 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:18.549745 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:18.549753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:18.550294 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:18.550890 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:18.550908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.552451 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.552455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:18.554106 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:18.554122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:18.555108 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:18.555116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:18.555341 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:18.556022 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:18.556043 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:18.558360 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:18.561928 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:18.561936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:18.561953 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:18.562417 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:18.562790 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:18.562793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:18.562795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:18.562997 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:18.563588 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:18.563594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:18.563953 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 16:48:18.564205 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 16:48:18.564765 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":272,"threads_started":1}
I20260928 16:48:18.567615 12277 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37553
I20260928 16:48:18.567931 12277 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:18.568735 12277 heartbeater.cc:511] Master 127.26.101.62:37553 requested a full tablet report, sending...
I20260928 16:48:18.570235 11949 ts_manager.cc:194] Registered new tserver with Master: f08d2d604a2f4c86a850425b70714889 (127.26.101.2:33595)
I20260928 16:48:18.571058 11949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:52517
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:48:18.611130 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:18.948966 12283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:18.949273 12283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:18.949369 12283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:18.956866 12283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:18.956996 12283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:18.957109 12283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:18.967278 12283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:18.968732 12283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:18.970114 12283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:18.983599 12289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:18.983660 12291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:18.983628 12288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:18.984544 12283 server_base.cc:1034] running on GCE node
I20260928 16:48:18.985248 12283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:18.986277 12283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:18.987479 12283 hybrid_clock.cc:648] HybridClock initialized: now 1790614098987425 us; error 44 us; skew 500 ppm
I20260928 16:48:18.987813 12283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:18 dist-test-slave-t0tw krb5kdc[11903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614098, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:18.991261 12283 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:48:18.992941 12283 webserver.cc:460] Webserver started at http://127.26.101.3:34375/ using document root <none> and password file <none>
I20260928 16:48:18.993667 12283 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:18.993798 12283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:18.994117 12283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:18.997238 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "04fb11170e3b488a890c676fc0d97737"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "e3864076f7ae2696589ea6f23a8ef8aa"
server_key_iv: "f747b3012853095f71eb3b0b195f058e"
server_key_version: "encryptionkey@0"
I20260928 16:48:18.998059 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "04fb11170e3b488a890c676fc0d97737"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "e3864076f7ae2696589ea6f23a8ef8aa"
server_key_iv: "f747b3012853095f71eb3b0b195f058e"
server_key_version: "encryptionkey@0"
I20260928 16:48:19.003700 12283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:48:19.007570 12298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:19.009061 12283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:19.009253 12283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "04fb11170e3b488a890c676fc0d97737"
format_stamp: "Formatted at 2026-09-28 16:48:18 on dist-test-slave-t0tw"
server_key: "e3864076f7ae2696589ea6f23a8ef8aa"
server_key_iv: "f747b3012853095f71eb3b0b195f058e"
server_key_version: "encryptionkey@0"
I20260928 16:48:19.009462 12283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:19.030622 12283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:19.034984 12283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:19.035317 12283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:19.036302 12283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:19.037714 12283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:19.037793 12283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:19.037927 12283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:19.037983 12283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:19.053463 12141 heartbeater.cc:503] Master 127.26.101.62:37553 was elected leader, sending a full tablet report...
I20260928 16:48:19.072588 12283 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:43915
I20260928 16:48:19.072654 12411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:43915 every 8 connection(s)
I20260928 16:48:19.074092 12283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:19.075975 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12283
I20260928 16:48:19.076187 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:19.076664 27028 external_mini_cluster.cc:1428] Setting key c9ac6a5cdd840cbc72b48cd810a4d280
Sep 28 16:48:19 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614098, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:19.096014 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.077026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:34863 (local address 127.26.101.62:37553)
0928 16:48:19.077550 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.077556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.078652 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 16:48:19.080640 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.080645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.080733 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.080838 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:19.082423 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.083305 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.084954 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.085394 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.085515 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.089192 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:19.089198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:19.089201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:19.089240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:19.092688 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.093518 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.093523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.093525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.093661 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.094124 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.094127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.094129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.094434 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:19.094583 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.095370 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.095616 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":231,"threads_started":1}
I20260928 16:48:19.097016 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.077611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37553 (local address 127.26.101.3:34863)
0928 16:48:19.078481 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.078545 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.080375 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.081085 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.081094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.081659 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:19.082240 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.082259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.083460 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.083465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.084745 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.084759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.085650 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.085659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.085898 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.086587 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:19.086617 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:19.088936 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:19.092864 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:19.092875 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:19.092891 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:19.093389 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:19.093778 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:19.093783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:19.093787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:19.094009 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:19.094567 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:19.094576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:19.094961 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.095265 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.095872 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":306,"threads_started":1}
I20260928 16:48:19.098556 12412 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37553
I20260928 16:48:19.098912 12412 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:19.099762 12412 heartbeater.cc:511] Master 127.26.101.62:37553 requested a full tablet report, sending...
I20260928 16:48:19.101400 11949 ts_manager.cc:194] Registered new tserver with Master: 04fb11170e3b488a890c676fc0d97737 (127.26.101.3:43915)
I20260928 16:48:19.102274 11949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:34863
I20260928 16:48:19.105022 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:19.123674 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.109658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.26.101.62:37553)
0928 16:48:19.109900 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.109907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.110136 (+ 229us) server_negotiation.cc:407] Connection header received
0928 16:48:19.110282 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.110286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.110356 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.110445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:19.111998 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.112736 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.114207 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.114688 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.114794 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.117293 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:19.117298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:19.117302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:19.117344 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:19.120373 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.121374 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.121381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.121385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.121605 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.122089 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.122094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.122098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.122581 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:19.122807 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.123207 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.123439 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 16:48:19.127933 11949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:19.135241 11949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:19.161089 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.149551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.26.101.1:39497)
0928 16:48:19.152230 (+ 2679us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.152239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.152268 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:48:19.152376 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.152383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.152662 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.152857 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:19.153499 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.155996 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.158305 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.159838 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.159988 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.160236 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.160566 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.160749 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2525,"thread_start_us":2364,"threads_started":1}
I20260928 16:48:19.161226 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.148870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33595 (local address 127.0.0.1:44988)
0928 16:48:19.149633 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.149680 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.149880 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.153918 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.153923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.153950 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:19.154381 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.154395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.156288 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.156294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.158515 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.158536 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.160473 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.160480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.160658 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.160689 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.160764 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.160900 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":649,"threads_started":1}
I20260928 16:48:19.161249 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.148956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:39497 (local address 127.0.0.1:58506)
0928 16:48:19.151959 (+ 3003us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.151978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.152108 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.152909 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.152916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.152949 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:19.153321 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.153330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.156209 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.156218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.158032 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.158047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.160081 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.160088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.160291 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.160340 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.160508 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.160723 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2865,"mutex_wait_us":1,"thread_start_us":2688,"threads_started":1}
I20260928 16:48:19.162169 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.149350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.26.101.2:33595)
0928 16:48:19.151547 (+ 2197us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.151558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.151592 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:48:19.152294 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.152308 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.153722 (+ 1414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.153940 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:19.154605 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.156140 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.158712 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.160016 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.160149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.160329 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.161692 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.161867 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046,"thread_start_us":1922,"threads_started":1}
I20260928 16:48:19.163478 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.152019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43915 (local address 127.0.0.1:56406)
0928 16:48:19.154862 (+ 2843us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.154884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.155002 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.155882 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.155886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.155910 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:19.156284 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.156296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.158471 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.158476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.161127 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.161143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.162729 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.162736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.162901 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.162932 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.163021 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.163175 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2727,"thread_start_us":2587,"threads_started":1}
I20260928 16:48:19.164161 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.152593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.26.101.3:43915)
0928 16:48:19.154588 (+ 1995us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.154595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.155143 (+ 548us) server_negotiation.cc:407] Connection header received
0928 16:48:19.155245 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.155253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.155683 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.155872 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:19.156474 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.158036 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.161336 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.162525 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.162660 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.162920 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.163748 (+ 828us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.163922 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1822,"thread_start_us":1658,"threads_started":1}
I20260928 16:48:19.166019 12211 tablet_service.cc:1467] Processing CreateTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 (DEFAULT_TABLE table=test-table [id=7f923d3a3972499981358cb001128091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:19.166019 12075 tablet_service.cc:1467] Processing CreateTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 (DEFAULT_TABLE table=test-table [id=7f923d3a3972499981358cb001128091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:19.167601 12075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac7e76b0f3c459b9f7d5778b8f994a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:19.167608 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac7e76b0f3c459b9f7d5778b8f994a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:19.167467 12346 tablet_service.cc:1467] Processing CreateTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 (DEFAULT_TABLE table=test-table [id=7f923d3a3972499981358cb001128091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:19.168704 12346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac7e76b0f3c459b9f7d5778b8f994a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:19.182327 12433 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Bootstrap starting.
I20260928 16:48:19.184811 12431 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Bootstrap starting.
I20260928 16:48:19.187428 12433 tablet_bootstrap.cc:654] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:19.188486 12433 log.cc:824] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:19.188868 12431 tablet_bootstrap.cc:654] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:19.189613 12432 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Bootstrap starting.
I20260928 16:48:19.190201 12431 log.cc:824] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:19.193153 12432 tablet_bootstrap.cc:654] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:19.193608 12433 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: No bootstrap required, opened a new log
I20260928 16:48:19.193608 12431 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: No bootstrap required, opened a new log
I20260928 16:48:19.193912 12431 ts_tablet_manager.cc:1397] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 16:48:19.193912 12433 ts_tablet_manager.cc:1397] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260928 16:48:19.194347 12432 log.cc:824] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:19.196982 12432 tablet_bootstrap.cc:492] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: No bootstrap required, opened a new log
I20260928 16:48:19.197230 12432 ts_tablet_manager.cc:1397] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:48:19.204535 12432 raft_consensus.cc:348] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.204587 12431 raft_consensus.cc:348] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.204583 12433 raft_consensus.cc:348] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.204980 12432 raft_consensus.cc:374] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:19.205024 12433 raft_consensus.cc:374] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:19.205024 12431 raft_consensus.cc:374] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:19.205106 12432 raft_consensus.cc:729] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08d2d604a2f4c86a850425b70714889, State: Initialized, Role: FOLLOWER
I20260928 16:48:19.205127 12431 raft_consensus.cc:729] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fb11170e3b488a890c676fc0d97737, State: Initialized, Role: FOLLOWER
I20260928 16:48:19.205179 12433 raft_consensus.cc:729] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4aa454a9c74579ab9ff32cb756bf8c, State: Initialized, Role: FOLLOWER
I20260928 16:48:19.205854 12432 consensus_queue.cc:260] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.205826 12433 consensus_queue.cc:260] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.205828 12431 consensus_queue.cc:260] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.207100 12412 heartbeater.cc:503] Master 127.26.101.62:37553 was elected leader, sending a full tablet report...
I20260928 16:48:19.207674 12277 heartbeater.cc:503] Master 127.26.101.62:37553 was elected leader, sending a full tablet report...
I20260928 16:48:19.207803 12431 ts_tablet_manager.cc:1428] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 16:48:19.210316 12432 ts_tablet_manager.cc:1428] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260928 16:48:19.211992 12433 ts_tablet_manager.cc:1428] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
W20260928 16:48:19.271500 12142 tablet.cc:2369] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:19.282271 12439 raft_consensus.cc:482] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:19.282634 12439 raft_consensus.cc:504] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.284488 12439 leader_election.cc:288] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f08d2d604a2f4c86a850425b70714889 (127.26.101.2:33595), 04fb11170e3b488a890c676fc0d97737 (127.26.101.3:43915)
I20260928 16:48:19.292551 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.285206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:57517 (local address 127.26.101.3:43915)
0928 16:48:19.285412 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.285418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.285949 (+ 531us) server_negotiation.cc:407] Connection header received
0928 16:48:19.286091 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.286095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.286153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.286220 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:19.286795 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.287805 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.290183 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.291389 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.291572 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.291657 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.292205 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.292326 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:19.292551 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.284838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:48199 (local address 127.26.101.2:33595)
0928 16:48:19.285047 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.285053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.285558 (+ 505us) server_negotiation.cc:407] Connection header received
0928 16:48:19.285774 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.285778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.285843 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.285929 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:19.286484 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.287538 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.290131 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.291447 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.291582 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.291707 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.292123 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.292250 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:48:19.292706 12440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.284694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33595 (local address 127.26.101.1:48199)
0928 16:48:19.285410 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.285434 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.285609 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.285966 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.285970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.285992 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:19.286334 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.286349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.287669 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.287673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.289959 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.289977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.291640 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.291646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.291932 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.291981 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.292094 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.292241 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":95,"thread_start_us":377,"threads_started":1}
I20260928 16:48:19.292743 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.285001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43915 (local address 127.26.101.1:57517)
0928 16:48:19.285842 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:19.285855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:19.285962 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:19.286267 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:19.286274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:19.286301 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:19.286645 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.286654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.287927 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.287932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.290046 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:19.290058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.291628 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.291636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:19.291926 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.291986 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 16:48:19.292163 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 16:48:19.292265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":139,"thread_start_us":544,"threads_started":1}
I20260928 16:48:19.293766 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac7e76b0f3c459b9f7d5778b8f994a8" candidate_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08d2d604a2f4c86a850425b70714889" is_pre_election: true
I20260928 16:48:19.293807 12366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac7e76b0f3c459b9f7d5778b8f994a8" candidate_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fb11170e3b488a890c676fc0d97737" is_pre_election: true
I20260928 16:48:19.294265 12231 raft_consensus.cc:2454] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a4aa454a9c74579ab9ff32cb756bf8c in term 0.
I20260928 16:48:19.294283 12366 raft_consensus.cc:2454] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a4aa454a9c74579ab9ff32cb756bf8c in term 0.
I20260928 16:48:19.294898 12029 leader_election.cc:302] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fb11170e3b488a890c676fc0d97737, 7a4aa454a9c74579ab9ff32cb756bf8c; no voters:
W20260928 16:48:19.295329 12278 tablet.cc:2369] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:19.295334 12439 raft_consensus.cc:2781] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:19.295499 12439 raft_consensus.cc:482] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:19.295609 12439 raft_consensus.cc:3037] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:19.298354 12439 raft_consensus.cc:504] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.298957 12439 leader_election.cc:288] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [CANDIDATE]: Term 1 election: Requested vote from peers f08d2d604a2f4c86a850425b70714889 (127.26.101.2:33595), 04fb11170e3b488a890c676fc0d97737 (127.26.101.3:43915)
I20260928 16:48:19.299523 12231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac7e76b0f3c459b9f7d5778b8f994a8" candidate_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08d2d604a2f4c86a850425b70714889"
I20260928 16:48:19.299724 12231 raft_consensus.cc:3037] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:19.299650 12366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac7e76b0f3c459b9f7d5778b8f994a8" candidate_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fb11170e3b488a890c676fc0d97737"
I20260928 16:48:19.299907 12366 raft_consensus.cc:3037] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:19.301944 12231 raft_consensus.cc:2454] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4aa454a9c74579ab9ff32cb756bf8c in term 1.
I20260928 16:48:19.302127 12366 raft_consensus.cc:2454] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4aa454a9c74579ab9ff32cb756bf8c in term 1.
I20260928 16:48:19.302402 12028 leader_election.cc:302] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a4aa454a9c74579ab9ff32cb756bf8c, f08d2d604a2f4c86a850425b70714889; no voters:
I20260928 16:48:19.302726 12439 raft_consensus.cc:2781] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:19.303227 12439 raft_consensus.cc:686] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 1 LEADER]: Becoming Leader. State: Replica: 7a4aa454a9c74579ab9ff32cb756bf8c, State: Running, Role: LEADER
I20260928 16:48:19.303757 12439 consensus_queue.cc:237] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } }
I20260928 16:48:19.308244 11947 catalog_manager.cc:5495] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a4aa454a9c74579ab9ff32cb756bf8c (127.26.101.1). New cstate: current_term: 1 leader_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a4aa454a9c74579ab9ff32cb756bf8c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 39497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 } health_report { overall_health: UNKNOWN } } }
W20260928 16:48:19.325825 12413 tablet.cc:2369] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:19.357280 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.351705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.26.101.1:39497)
0928 16:48:19.351932 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.351938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.351957 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:48:19.352190 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.352194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.352292 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.352423 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:19.352820 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.353558 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.355024 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.355490 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.355605 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.356145 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:19.356666 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:19.356803 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.356927 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.357018 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:48:19.365715 12366 raft_consensus.cc:1261] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 1 FOLLOWER]: Refusing update from remote peer 7a4aa454a9c74579ab9ff32cb756bf8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:19.365748 12231 raft_consensus.cc:1261] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 1 FOLLOWER]: Refusing update from remote peer 7a4aa454a9c74579ab9ff32cb756bf8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:19.366700 12442 consensus_queue.cc:1035] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fb11170e3b488a890c676fc0d97737" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:19.367057 12439 consensus_queue.cc:1035] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08d2d604a2f4c86a850425b70714889" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:19.379756 12445 mvcc.cc:205] Tried to move back new op lower bound from 7334355350992121856 to 7334355350752055296. Current Snapshot: MvccSnapshot[applied={T|T < 7334355350992121856}]
I20260928 16:48:19.384178 12447 mvcc.cc:205] Tried to move back new op lower bound from 7334355350992121856 to 7334355350752055296. Current Snapshot: MvccSnapshot[applied={T|T < 7334355350992121856}]
I20260928 16:48:19.393052 11947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:19.393319 11947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:19.396876 11947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b5e6893c4c34b06a4580db9e411c521: Sending DeleteTablet for 3 replicas of tablet fac7e76b0f3c459b9f7d5778b8f994a8
I20260928 16:48:19.398013 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:19 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:44988
I20260928 16:48:19.398051 12075 tablet_service.cc:1514] Processing DeleteTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:19 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:58506
I20260928 16:48:19.398741 12346 tablet_service.cc:1514] Processing DeleteTablet for tablet fac7e76b0f3c459b9f7d5778b8f994a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:19 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:56406
I20260928 16:48:19.400848 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12010
I20260928 16:48:19.410883 12455 tablet_replica.cc:330] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: stopping tablet replica
I20260928 16:48:19.411561 12455 raft_consensus.cc:2229] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:19.412184 12455 raft_consensus.cc:2258] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:19.413053 12454 tablet_replica.cc:330] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: stopping tablet replica
I20260928 16:48:19.413626 12454 raft_consensus.cc:2229] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:19.414480 12454 raft_consensus.cc:2258] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:19.415496 12455 ts_tablet_manager.cc:1905] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:19.416339 12454 ts_tablet_manager.cc:1905] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:19.416589 12456 tablet_replica.cc:330] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: stopping tablet replica
I20260928 16:48:19.418135 12456 raft_consensus.cc:2229] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:19.422232 12456 raft_consensus.cc:2258] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:19.423820 12454 ts_tablet_manager.cc:1918] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:19.424036 12454 log.cc:1196] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/fac7e76b0f3c459b9f7d5778b8f994a8
I20260928 16:48:19.424611 12454 ts_tablet_manager.cc:1939] T fac7e76b0f3c459b9f7d5778b8f994a8 P 7a4aa454a9c74579ab9ff32cb756bf8c: Deleting consensus metadata
I20260928 16:48:19.425319 12456 ts_tablet_manager.cc:1905] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:19.426446 11933 catalog_manager.cc:4841] TS 7a4aa454a9c74579ab9ff32cb756bf8c (127.26.101.1:39497): tablet fac7e76b0f3c459b9f7d5778b8f994a8 (table test-table [id=7f923d3a3972499981358cb001128091]) successfully deleted
I20260928 16:48:19.433015 12456 ts_tablet_manager.cc:1918] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:19.433293 12456 log.cc:1196] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/fac7e76b0f3c459b9f7d5778b8f994a8
I20260928 16:48:19.433476 12455 ts_tablet_manager.cc:1918] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:19.433718 12455 log.cc:1196] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/fac7e76b0f3c459b9f7d5778b8f994a8
I20260928 16:48:19.433928 12456 ts_tablet_manager.cc:1939] T fac7e76b0f3c459b9f7d5778b8f994a8 P 04fb11170e3b488a890c676fc0d97737: Deleting consensus metadata
Sep 28 16:48:19 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:48:19.434319 12455 ts_tablet_manager.cc:1939] T fac7e76b0f3c459b9f7d5778b8f994a8 P f08d2d604a2f4c86a850425b70714889: Deleting consensus metadata
I20260928 16:48:19.436058 11936 catalog_manager.cc:4841] TS 04fb11170e3b488a890c676fc0d97737 (127.26.101.3:43915): tablet fac7e76b0f3c459b9f7d5778b8f994a8 (table test-table [id=7f923d3a3972499981358cb001128091]) successfully deleted
I20260928 16:48:19.436105 11935 catalog_manager.cc:4841] TS f08d2d604a2f4c86a850425b70714889 (127.26.101.2:33595): tablet fac7e76b0f3c459b9f7d5778b8f994a8 (table test-table [id=7f923d3a3972499981358cb001128091]) successfully deleted
I20260928 16:48:19.441965 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.402308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.26.101.1:39497)
0928 16:48:19.402491 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.402497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.406493 (+ 3996us) server_negotiation.cc:407] Connection header received
0928 16:48:19.406662 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.406668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.406753 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.406868 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:19.418445 (+ 11577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.419806 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.422969 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.423677 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.423779 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.435709 (+ 11930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:19.435731 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:19.435748 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:19.435805 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:19.438996 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.439783 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.439791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.439794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.439958 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.440357 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.440361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.440363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.440737 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:19.441044 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.441372 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.441635 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:48:19.442643 12136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:19.551234 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12010
I20260928 16:48:19.568385 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12147
Sep 28 16:48:19 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:48:19.583154 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.569387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.26.101.2:33595)
0928 16:48:19.569569 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.569574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.569592 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:48:19.569649 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.569653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.569713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.569795 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:19.571228 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.572036 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.573219 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.573645 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.573736 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.577766 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:19.577782 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:19.577800 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:19.577842 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:19.580659 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.581348 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.581356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.581358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.581489 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.581786 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.581789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.581791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.582174 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:19.582432 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.582637 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.582803 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:48:19.583855 12272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:19.672633 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12147
I20260928 16:48:19.688071 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12283
Sep 28 16:48:19 dist-test-slave-t0tw krb5kdc[11903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:48:19.704586 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:19.688886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.26.101.3:43915)
0928 16:48:19.689079 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:19.689084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:19.689104 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:19.689240 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:19.689244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:19.689329 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:19.689418 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:19.690792 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.691616 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.692966 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:19.693396 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:19.693508 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:19.698283 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:19.698301 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:19.698317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:19.698367 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:19.701259 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.702076 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.702084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.702087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.702264 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:19.702704 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:19.702708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:19.702710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:19.703324 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:19.703648 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:19.704003 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 16:48:19.704222 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:48:19.705313 12407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:19.795449 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12283
I20260928 16:48:19.811033 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 11916
I20260928 16:48:19.812234 11979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:19.924839 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 11916
2026-09-28T16:48:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4264 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T16:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:19Z Disabled control of system clock
I20260928 16:48:20.004122 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40907
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:40907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:20.348728 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:20.349097 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:20.349228 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:20.356511 12465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:20.356639 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:20.356696 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:20.356776 12465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:20.356833 12465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:20.366221 12465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:40907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:20.367606 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:20.369526 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:20.383131 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:20.383020 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:20.383000 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:20.384138 12465 server_base.cc:1034] running on GCE node
I20260928 16:48:20.385039 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:20.386597 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:20.387832 12465 hybrid_clock.cc:648] HybridClock initialized: now 1790614100387782 us; error 39 us; skew 500 ppm
I20260928 16:48:20.388224 12465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:20.390882 12465 webserver.cc:460] Webserver started at http://127.26.101.62:41747/ using document root <none> and password file <none>
I20260928 16:48:20.391664 12465 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:20.391793 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:20.392118 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:20.395370 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "52074e2e932b4b6da58b15f6c157b548"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "0cd4dad3c951906b57614df880ef2be5"
server_key_iv: "c81562f4905bcfe1d5adc8046993d89c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.396217 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "52074e2e932b4b6da58b15f6c157b548"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "0cd4dad3c951906b57614df880ef2be5"
server_key_iv: "c81562f4905bcfe1d5adc8046993d89c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.401943 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:20.406106 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:20.407604 12465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:20.407790 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "52074e2e932b4b6da58b15f6c157b548"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "0cd4dad3c951906b57614df880ef2be5"
server_key_iv: "c81562f4905bcfe1d5adc8046993d89c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.408012 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:20.423971 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:20.424898 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:20.425191 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:20.444552 12465 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:40907
I20260928 16:48:20.444576 12531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:40907 every 8 connection(s)
I20260928 16:48:20.446094 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:20.450367 12532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:20.451612 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12465
I20260928 16:48:20.451802 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:20.452394 27028 external_mini_cluster.cc:1428] Setting key 26fef0f9e37bba417d4b67d2aac501cf
I20260928 16:48:20.464778 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: Bootstrap starting.
I20260928 16:48:20.467126 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:20.454893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.26.101.62:40907)
0928 16:48:20.456092 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:20.456108 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:20.456159 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:48:20.457879 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:20.457908 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:20.458504 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:20.458935 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:20.459596 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:20.460909 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:20.462349 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:20.463159 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:20.463348 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:20.464601 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:20.464610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:20.464626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:20.465252 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:20.465350 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:20.465374 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:20.465480 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:20.465711 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:48:20.466067 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":386,"threads_started":1}
I20260928 16:48:20.468335 12532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:20.469452 12532 log.cc:824] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:20.472889 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: No bootstrap required, opened a new log
I20260928 16:48:20.481029 12532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } }
I20260928 16:48:20.481472 12532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:20.481588 12532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52074e2e932b4b6da58b15f6c157b548, State: Initialized, Role: FOLLOWER
I20260928 16:48:20.482211 12532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } }
I20260928 16:48:20.482445 12532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:20.482590 12532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:20.482755 12532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:20.485349 12532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } }
I20260928 16:48:20.485970 12532 leader_election.cc:302] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52074e2e932b4b6da58b15f6c157b548; no voters:
I20260928 16:48:20.486614 12532 leader_election.cc:288] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:20.486815 12537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:20.487629 12537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [term 1 LEADER]: Becoming Leader. State: Replica: 52074e2e932b4b6da58b15f6c157b548, State: Running, Role: LEADER
I20260928 16:48:20.488174 12537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } }
I20260928 16:48:20.488770 12532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:20.491541 12538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52074e2e932b4b6da58b15f6c157b548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } } }
I20260928 16:48:20.491600 12539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52074e2e932b4b6da58b15f6c157b548. Latest consensus state: current_term: 1 leader_uuid: "52074e2e932b4b6da58b15f6c157b548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52074e2e932b4b6da58b15f6c157b548" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40907 } } }
I20260928 16:48:20.492031 12538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:20.492036 12539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:20.494731 12544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:20.499898 12544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:20.510342 12544 catalog_manager.cc:1347] Generated new cluster ID: 93bfc5cea7134bc09adc5409eb6fc451
I20260928 16:48:20.510500 12544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:20.544023 12544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:20.545382 12544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:20.558171 12544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: Generated new TSK 0
I20260928 16:48:20.559158 12544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:20.564071 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--builtin_ntp_servers=127.26.101.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:20.912518 12556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:20.912818 12556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:20.912938 12556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:20.920444 12556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:20.920574 12556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:20.920713 12556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:20.930177 12556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:20.931573 12556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:20.933390 12556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:20.946635 12561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:20.946641 12564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:20.946659 12562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:20.947675 12556 server_base.cc:1034] running on GCE node
I20260928 16:48:20.948338 12556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:20.949308 12556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:20.950529 12556 hybrid_clock.cc:648] HybridClock initialized: now 1790614100950480 us; error 34 us; skew 500 ppm
I20260928 16:48:20.950865 12556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:20.953505 12556 webserver.cc:460] Webserver started at http://127.26.101.1:45873/ using document root <none> and password file <none>
I20260928 16:48:20.954282 12556 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:20.954388 12556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:20.954695 12556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:20.957646 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "eae8a616d8c642de8fbddb5f58c6eec9"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "1a70e3045d96541e89777d1071ec3d8c"
server_key_iv: "1ed5d588696219bf471baa89b2e62f2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.958460 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "eae8a616d8c642de8fbddb5f58c6eec9"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "1a70e3045d96541e89777d1071ec3d8c"
server_key_iv: "1ed5d588696219bf471baa89b2e62f2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.963912 12556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:48:20.967769 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:20.969277 12556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:20.969444 12556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "eae8a616d8c642de8fbddb5f58c6eec9"
format_stamp: "Formatted at 2026-09-28 16:48:20 on dist-test-slave-t0tw"
server_key: "1a70e3045d96541e89777d1071ec3d8c"
server_key_iv: "1ed5d588696219bf471baa89b2e62f2c"
server_key_version: "encryptionkey@0"
I20260928 16:48:20.969676 12556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:20.990631 12556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:20.991591 12556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:20.991902 12556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:20.992969 12556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:20.994521 12556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:20.994616 12556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:20.994724 12556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:20.994797 12556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.024395 12556 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:45405
I20260928 16:48:21.024479 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:45405 every 8 connection(s)
I20260928 16:48:21.025717 12556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:21.033962 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12556
I20260928 16:48:21.034168 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:21.034744 27028 external_mini_cluster.cc:1428] Setting key 305ac92e77bc7e34a35d573a5bc617a6
I20260928 16:48:21.038038 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--builtin_ntp_servers=127.26.101.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:21.039259 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.028267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:33191 (local address 127.26.101.62:40907)
0928 16:48:21.028716 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:21.028723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:21.030003 (+ 1280us) server_negotiation.cc:407] Connection header received
0928 16:48:21.031793 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:21.031797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:21.031867 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:21.031964 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:21.033029 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.033737 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.035962 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.036552 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.036671 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.038277 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:21.038286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:21.038291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:21.038318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:21.038378 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:21.038397 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:21.038481 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:21.038771 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 16:48:21.038860 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":198,"threads_started":1}
I20260928 16:48:21.040493 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.028850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40907 (local address 127.26.101.1:33191)
0928 16:48:21.029843 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.029884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:21.031558 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:21.032144 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:21.032152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:21.032239 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:21.032865 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.032882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.033921 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.033927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.035795 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.035822 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.036766 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.036772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.037061 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.037824 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:21.037828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:21.037886 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:21.037999 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:21.038503 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:21.038515 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:21.038523 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:21.038715 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:48:21.039182 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":272,"threads_started":1}
I20260928 16:48:21.042315 12684 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40907
I20260928 16:48:21.042630 12684 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:21.043454 12684 heartbeater.cc:511] Master 127.26.101.62:40907 requested a full tablet report, sending...
I20260928 16:48:21.045739 12496 ts_manager.cc:194] Registered new tserver with Master: eae8a616d8c642de8fbddb5f58c6eec9 (127.26.101.1:45405)
I20260928 16:48:21.047442 12496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.1:33191
W20260928 16:48:21.374470 12688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:21.374759 12688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:21.374889 12688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:21.382129 12688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:21.382232 12688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:21.382375 12688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:21.391469 12688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:21.392819 12688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:21.394659 12688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:21.407300 12693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:21.407285 12696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:21.407328 12694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:21.408299 12688 server_base.cc:1034] running on GCE node
I20260928 16:48:21.408879 12688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:21.409853 12688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:21.411095 12688 hybrid_clock.cc:648] HybridClock initialized: now 1790614101411022 us; error 53 us; skew 500 ppm
I20260928 16:48:21.411440 12688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:21.413724 12688 webserver.cc:460] Webserver started at http://127.26.101.2:44809/ using document root <none> and password file <none>
I20260928 16:48:21.414395 12688 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:21.414499 12688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:21.414793 12688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:21.417686 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "79006f8409014e7d965fca54a7b1627e"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "96b21188864682eef480d239a35804f4"
server_key_iv: "f620844ac9a6b6a74131e1fcec06ea21"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.418493 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "79006f8409014e7d965fca54a7b1627e"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "96b21188864682eef480d239a35804f4"
server_key_iv: "f620844ac9a6b6a74131e1fcec06ea21"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.423696 12688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 16:48:21.427140 12702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.428416 12688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:21.428571 12688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "79006f8409014e7d965fca54a7b1627e"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "96b21188864682eef480d239a35804f4"
server_key_iv: "f620844ac9a6b6a74131e1fcec06ea21"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.428777 12688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:21.443691 12688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:21.444576 12688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:21.444856 12688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:21.445839 12688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:21.447325 12688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:21.447422 12688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.447533 12688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:21.447607 12688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.478447 12688 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:33755
I20260928 16:48:21.478489 12814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:33755 every 8 connection(s)
I20260928 16:48:21.479746 12688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:21.483949 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12688
I20260928 16:48:21.484131 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:21.484668 27028 external_mini_cluster.cc:1428] Setting key bc983ba2ac6ca8c4deaaf81389722ede
I20260928 16:48:21.487792 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--builtin_ntp_servers=127.26.101.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:21.493567 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.482310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:58751 (local address 127.26.101.62:40907)
0928 16:48:21.482516 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:21.482521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:21.483905 (+ 1384us) server_negotiation.cc:407] Connection header received
0928 16:48:21.486121 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:21.486127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:21.486202 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:21.486306 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:21.487662 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.488496 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.490660 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.491067 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.491209 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.492766 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:21.492771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:21.492775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:21.492797 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:21.492834 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:21.492848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:21.492929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:21.493205 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 16:48:21.493307 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:48:21.494944 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.482773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40907 (local address 127.26.101.2:58751)
0928 16:48:21.483724 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.483772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:21.485866 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:21.486537 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:21.486549 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:21.486642 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:21.487464 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.487493 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.488665 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.488671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.490520 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.490535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.491328 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.491335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.491619 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.492358 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:21.492361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:21.492422 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:21.492526 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:21.492944 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:21.492953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:21.492961 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:21.493151 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 16:48:21.493598 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":281,"threads_started":1}
I20260928 16:48:21.496469 12815 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40907
I20260928 16:48:21.496776 12815 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:21.497563 12815 heartbeater.cc:511] Master 127.26.101.62:40907 requested a full tablet report, sending...
I20260928 16:48:21.499028 12496 ts_manager.cc:194] Registered new tserver with Master: 79006f8409014e7d965fca54a7b1627e (127.26.101.2:33755)
I20260928 16:48:21.499854 12496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.2:58751
W20260928 16:48:21.819796 12818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:21.820104 12818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:21.820236 12818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:21.827594 12818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:21.827721 12818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:21.827873 12818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:21.837133 12818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:21.838456 12818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:21.840224 12818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:21.853538 12823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:21.853507 12824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:21.853569 12826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:21.854388 12818 server_base.cc:1034] running on GCE node
I20260928 16:48:21.855062 12818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:21.856118 12818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:21.857326 12818 hybrid_clock.cc:648] HybridClock initialized: now 1790614101857270 us; error 51 us; skew 500 ppm
I20260928 16:48:21.857640 12818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:21.860057 12818 webserver.cc:460] Webserver started at http://127.26.101.3:38917/ using document root <none> and password file <none>
I20260928 16:48:21.860764 12818 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:21.860868 12818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:21.861168 12818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:21.864250 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "a6c4490dc2694d4baaecb69467509194"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "d6b7d0bb21d03b4abc2fb6d0bc59fb89"
server_key_iv: "fe9da3a6d6df5e55335fadc198764bae"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.865036 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "a6c4490dc2694d4baaecb69467509194"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "d6b7d0bb21d03b4abc2fb6d0bc59fb89"
server_key_iv: "fe9da3a6d6df5e55335fadc198764bae"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.870146 12818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:21.873759 12832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.875172 12818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:21.875315 12818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "a6c4490dc2694d4baaecb69467509194"
format_stamp: "Formatted at 2026-09-28 16:48:21 on dist-test-slave-t0tw"
server_key: "d6b7d0bb21d03b4abc2fb6d0bc59fb89"
server_key_iv: "fe9da3a6d6df5e55335fadc198764bae"
server_key_version: "encryptionkey@0"
I20260928 16:48:21.875499 12818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:21.907855 12818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:21.908696 12818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:21.908970 12818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:21.909935 12818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:21.911424 12818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:21.911576 12818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.911666 12818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:21.911721 12818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:21.942438 12818 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:39773
I20260928 16:48:21.942490 12946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:39773 every 8 connection(s)
I20260928 16:48:21.943810 12818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:21.945773 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 12818
I20260928 16:48:21.945937 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:21.946475 27028 external_mini_cluster.cc:1428] Setting key fc9dfa910bfa116096059cfa9673d1a3
I20260928 16:48:21.957017 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.946570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:39587 (local address 127.26.101.62:40907)
0928 16:48:21.946788 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:21.946795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:21.948492 (+ 1697us) server_negotiation.cc:407] Connection header received
0928 16:48:21.950375 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:21.950380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:21.950453 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:21.950547 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:21.951814 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.952596 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.954177 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.954585 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.954683 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.956172 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:21.956177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:21.956180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:21.956202 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:21.956235 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:21.956246 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:21.956313 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:21.956658 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 16:48:21.956761 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:48:21.957957 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.947230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40907 (local address 127.26.101.3:39587)
0928 16:48:21.948334 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:21.948382 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:21.950030 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:21.950764 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:21.950773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:21.950871 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:21.951620 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.951642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.952756 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.952761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.954048 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:21.954062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.954836 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.954842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:21.955066 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.955851 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:21.955853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:21.955897 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:21.955982 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:21.956339 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:21.956347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:21.956354 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:48:21.956516 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:48:21.956852 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":319,"threads_started":1}
I20260928 16:48:21.959440 12947 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40907
I20260928 16:48:21.959748 12947 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:21.960515 12947 heartbeater.cc:511] Master 127.26.101.62:40907 requested a full tablet report, sending...
I20260928 16:48:21.961903 12496 ts_manager.cc:194] Registered new tserver with Master: a6c4490dc2694d4baaecb69467509194 (127.26.101.3:39773)
I20260928 16:48:21.962767 12496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.3:39587
I20260928 16:48:21.974093 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:21.984123 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:21.978408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.26.101.62:40907)
0928 16:48:21.978646 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:21.978652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:21.978997 (+ 345us) server_negotiation.cc:407] Connection header received
0928 16:48:21.979190 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:21.979195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:21.979330 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:21.979456 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:21.979958 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.980668 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.982080 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:21.982480 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:21.982569 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:21.983544 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:21.983549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:21.983553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:21.983571 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:21.983605 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:21.983616 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:21.983683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:21.983798 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 16:48:21.983879 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 16:48:21.987852 12496 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:21.994984 12496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:22.019663 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.007601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33755 (local address 127.0.0.1:46246)
0928 16:48:22.009236 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.009287 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:22.009492 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:22.011482 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:22.011489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:22.011523 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:22.011964 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.011977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.014022 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.014027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.016538 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.016553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.018541 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.018547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.018803 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.018898 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 16:48:22.019027 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:48:22.019250 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":39,"thread_start_us":2359,"threads_started":1}
I20260928 16:48:22.020493 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.007873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.26.101.2:33755)
0928 16:48:22.008755 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.008766 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.009695 (+ 929us) server_negotiation.cc:407] Connection header received
0928 16:48:22.009812 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.009819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.010086 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.010287 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:22.012339 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.013857 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.016769 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.018000 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.018118 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.018412 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.019934 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.020130 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":608,"threads_started":1}
I20260928 16:48:22.021802 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.007958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47414)
0928 16:48:22.010256 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.010274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:22.011732 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:22.012720 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:22.012725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:22.012764 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:22.013149 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.013161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.016212 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.016216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.018881 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.018900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.020910 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.020916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.021124 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.021161 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:48:22.021264 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:48:22.021417 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"mutex_wait_us":470,"thread_start_us":861,"threads_started":1}
I20260928 16:48:22.021881 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.008590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.26.101.1:45405)
0928 16:48:22.009827 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.009836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.011616 (+ 1780us) server_negotiation.cc:407] Connection header received
0928 16:48:22.011952 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.011961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.012218 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.012380 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:22.013345 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.016042 (+ 2697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.019143 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.020362 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.020458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.020664 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.021336 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.021513 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":2107,"threads_started":1}
I20260928 16:48:22.024796 12750 tablet_service.cc:1467] Processing CreateTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 (DEFAULT_TABLE table=test-table [id=2e00dec24ef14c43b01615a95b690222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:22.025949 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.009942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:39773 (local address 127.0.0.1:36290)
0928 16:48:22.017361 (+ 7419us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.017379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:22.017511 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:22.018137 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:22.018141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:22.018167 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:22.018837 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.018850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.020800 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.020805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.022849 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.022866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.025155 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.025162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.025348 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.025383 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:48:22.025478 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:48:22.025619 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7289,"thread_start_us":7122,"threads_started":1}
I20260928 16:48:22.025866 12618 tablet_service.cc:1467] Processing CreateTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 (DEFAULT_TABLE table=test-table [id=2e00dec24ef14c43b01615a95b690222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:22.026713 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.010551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.26.101.3:39773)
0928 16:48:22.016896 (+ 6345us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.016914 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.017498 (+ 584us) server_negotiation.cc:407] Connection header received
0928 16:48:22.017669 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.017678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.017928 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.018125 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:22.019064 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.020608 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.023063 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.024595 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.024732 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.025018 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.026197 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.026380 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6157,"thread_start_us":5994,"threads_started":1}
I20260928 16:48:22.027493 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27bc9f513dfe48f6b738bee09d14f411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:22.027817 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27bc9f513dfe48f6b738bee09d14f411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:22.030789 12881 tablet_service.cc:1467] Processing CreateTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 (DEFAULT_TABLE table=test-table [id=2e00dec24ef14c43b01615a95b690222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:22.032330 12881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27bc9f513dfe48f6b738bee09d14f411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:22.043874 12965 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Bootstrap starting.
I20260928 16:48:22.046207 12966 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Bootstrap starting.
I20260928 16:48:22.047849 12967 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Bootstrap starting.
I20260928 16:48:22.048038 12965 tablet_bootstrap.cc:654] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:22.049232 12965 log.cc:824] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:22.050484 12684 heartbeater.cc:503] Master 127.26.101.62:40907 was elected leader, sending a full tablet report...
I20260928 16:48:22.051447 12966 tablet_bootstrap.cc:654] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:22.051473 12967 tablet_bootstrap.cc:654] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:22.052747 12965 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: No bootstrap required, opened a new log
I20260928 16:48:22.052770 12967 log.cc:824] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:22.052855 12966 log.cc:824] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:22.053153 12965 ts_tablet_manager.cc:1397] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 16:48:22.058243 12966 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: No bootstrap required, opened a new log
I20260928 16:48:22.058468 12967 tablet_bootstrap.cc:492] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: No bootstrap required, opened a new log
I20260928 16:48:22.058529 12966 ts_tablet_manager.cc:1397] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260928 16:48:22.058751 12967 ts_tablet_manager.cc:1397] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 16:48:22.063625 12965 raft_consensus.cc:348] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.064128 12965 raft_consensus.cc:374] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:22.064306 12965 raft_consensus.cc:729] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c4490dc2694d4baaecb69467509194, State: Initialized, Role: FOLLOWER
I20260928 16:48:22.065088 12965 consensus_queue.cc:260] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.066290 12947 heartbeater.cc:503] Master 127.26.101.62:40907 was elected leader, sending a full tablet report...
I20260928 16:48:22.066669 12966 raft_consensus.cc:348] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.067034 12965 ts_tablet_manager.cc:1428] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260928 16:48:22.067147 12966 raft_consensus.cc:374] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:22.067317 12966 raft_consensus.cc:729] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79006f8409014e7d965fca54a7b1627e, State: Initialized, Role: FOLLOWER
I20260928 16:48:22.067950 12966 consensus_queue.cc:260] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.069031 12815 heartbeater.cc:503] Master 127.26.101.62:40907 was elected leader, sending a full tablet report...
I20260928 16:48:22.068831 12967 raft_consensus.cc:348] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.069272 12967 raft_consensus.cc:374] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:22.069388 12967 raft_consensus.cc:729] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae8a616d8c642de8fbddb5f58c6eec9, State: Initialized, Role: FOLLOWER
I20260928 16:48:22.069725 12966 ts_tablet_manager.cc:1428] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 16:48:22.070022 12967 consensus_queue.cc:260] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.072445 12967 ts_tablet_manager.cc:1428] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260928 16:48:22.195590 12948 tablet.cc:2369] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:22.231840 12816 tablet.cc:2369] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:22.262012 12971 raft_consensus.cc:482] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:22.262383 12971 raft_consensus.cc:504] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.264225 12971 leader_election.cc:288] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79006f8409014e7d965fca54a7b1627e (127.26.101.2:33755), eae8a616d8c642de8fbddb5f58c6eec9 (127.26.101.1:45405)
I20260928 16:48:22.270020 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.264889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:52975 (local address 127.26.101.2:33755)
0928 16:48:22.265099 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.265106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.265127 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:22.265199 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.265203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.265273 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.265352 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:22.265857 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.266615 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.268349 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.269099 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.269256 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.269331 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.269603 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.269736 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:22.270022 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.264515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:33755 (local address 127.26.101.3:52975)
0928 16:48:22.264883 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.264914 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:22.265053 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:22.265402 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:22.265407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:22.265441 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:22.265685 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.265697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.266747 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.266753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.268177 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.268191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.269251 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.269257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.269446 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.269472 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:48:22.269550 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:48:22.269698 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1}
I20260928 16:48:22.271064 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.264869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:55427 (local address 127.26.101.1:45405)
0928 16:48:22.265116 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.265124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.265782 (+ 658us) server_negotiation.cc:407] Connection header received
0928 16:48:22.265949 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.265954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.266052 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.266141 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:22.266729 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.267475 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.269240 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.270047 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.270191 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.270264 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.270686 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.270819 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 16:48:22.271111 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.264612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.26.101.3:55427)
0928 16:48:22.265646 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.265662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:22.265810 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:22.266178 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:22.266182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:22.266206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:22.266561 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.266573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.267634 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.267638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.269054 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:22.269067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.270368 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.270374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:22.270529 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.270554 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:48:22.270633 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:48:22.270758 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":859,"threads_started":1}
I20260928 16:48:22.271010 12770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27bc9f513dfe48f6b738bee09d14f411" candidate_uuid: "a6c4490dc2694d4baaecb69467509194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79006f8409014e7d965fca54a7b1627e" is_pre_election: true
I20260928 16:48:22.271672 12770 raft_consensus.cc:2454] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6c4490dc2694d4baaecb69467509194 in term 0.
I20260928 16:48:22.272174 12638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27bc9f513dfe48f6b738bee09d14f411" candidate_uuid: "a6c4490dc2694d4baaecb69467509194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" is_pre_election: true
I20260928 16:48:22.272354 12837 leader_election.cc:302] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79006f8409014e7d965fca54a7b1627e, a6c4490dc2694d4baaecb69467509194; no voters:
I20260928 16:48:22.272629 12638 raft_consensus.cc:2454] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6c4490dc2694d4baaecb69467509194 in term 0.
I20260928 16:48:22.272759 12971 raft_consensus.cc:2781] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:22.272912 12971 raft_consensus.cc:482] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:22.273025 12971 raft_consensus.cc:3037] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:22.275367 12971 raft_consensus.cc:504] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.276036 12971 leader_election.cc:288] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [CANDIDATE]: Term 1 election: Requested vote from peers 79006f8409014e7d965fca54a7b1627e (127.26.101.2:33755), eae8a616d8c642de8fbddb5f58c6eec9 (127.26.101.1:45405)
I20260928 16:48:22.276588 12638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27bc9f513dfe48f6b738bee09d14f411" candidate_uuid: "a6c4490dc2694d4baaecb69467509194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae8a616d8c642de8fbddb5f58c6eec9"
I20260928 16:48:22.276589 12770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27bc9f513dfe48f6b738bee09d14f411" candidate_uuid: "a6c4490dc2694d4baaecb69467509194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79006f8409014e7d965fca54a7b1627e"
I20260928 16:48:22.276829 12638 raft_consensus.cc:3037] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:22.276832 12770 raft_consensus.cc:3037] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 0 FOLLOWER]: Advancing to term 1
W20260928 16:48:22.278476 12685 tablet.cc:2369] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:22.279307 12638 raft_consensus.cc:2454] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6c4490dc2694d4baaecb69467509194 in term 1.
I20260928 16:48:22.279609 12770 raft_consensus.cc:2454] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6c4490dc2694d4baaecb69467509194 in term 1.
I20260928 16:48:22.279760 12836 leader_election.cc:302] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6c4490dc2694d4baaecb69467509194, eae8a616d8c642de8fbddb5f58c6eec9; no voters:
I20260928 16:48:22.280105 12971 raft_consensus.cc:2781] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:22.280556 12971 raft_consensus.cc:686] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 1 LEADER]: Becoming Leader. State: Replica: a6c4490dc2694d4baaecb69467509194, State: Running, Role: LEADER
I20260928 16:48:22.281098 12971 consensus_queue.cc:237] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } }
I20260928 16:48:22.285766 12495 catalog_manager.cc:5495] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6c4490dc2694d4baaecb69467509194 (127.26.101.3). New cstate: current_term: 1 leader_uuid: "a6c4490dc2694d4baaecb69467509194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6c4490dc2694d4baaecb69467509194" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 39773 } health_report { overall_health: HEALTHY } } }
I20260928 16:48:22.318219 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.312938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.26.101.3:39773)
0928 16:48:22.313180 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.313185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.313205 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:22.313353 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.313357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.313435 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.313557 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:22.313898 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.314636 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.316113 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.316541 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.316638 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.317133 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:22.317633 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:22.317746 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.317874 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.317943 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:48:22.326448 12770 raft_consensus.cc:1261] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 1 FOLLOWER]: Refusing update from remote peer a6c4490dc2694d4baaecb69467509194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:22.326450 12638 raft_consensus.cc:1261] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9 [term 1 FOLLOWER]: Refusing update from remote peer a6c4490dc2694d4baaecb69467509194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:22.327399 12975 consensus_queue.cc:1035] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae8a616d8c642de8fbddb5f58c6eec9" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:22.327736 12971 consensus_queue.cc:1035] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79006f8409014e7d965fca54a7b1627e" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:22.339422 12978 mvcc.cc:205] Tried to move back new op lower bound from 7334355363119390720 to 7334355362947424256. Current Snapshot: MvccSnapshot[applied={T|T < 7334355363119390720}]
I20260928 16:48:22.341809 12980 mvcc.cc:205] Tried to move back new op lower bound from 7334355363119390720 to 7334355362947424256. Current Snapshot: MvccSnapshot[applied={T|T < 7334355363119390720}]
I20260928 16:48:22.354981 12496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:22.355374 12496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:22.359299 12496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52074e2e932b4b6da58b15f6c157b548: Sending DeleteTablet for 3 replicas of tablet 27bc9f513dfe48f6b738bee09d14f411
I20260928 16:48:22.360577 12881 tablet_service.cc:1514] Processing DeleteTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:22 UTC) from {username='slave'} at 127.0.0.1:36290
I20260928 16:48:22.360571 12618 tablet_service.cc:1514] Processing DeleteTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:22 UTC) from {username='slave'} at 127.0.0.1:47414
I20260928 16:48:22.360591 12750 tablet_service.cc:1514] Processing DeleteTablet for tablet 27bc9f513dfe48f6b738bee09d14f411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:22 UTC) from {username='slave'} at 127.0.0.1:46246
I20260928 16:48:22.362645 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12556
I20260928 16:48:22.395450 12989 tablet_replica.cc:330] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: stopping tablet replica
I20260928 16:48:22.396005 12989 raft_consensus.cc:2229] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:22.396781 12989 raft_consensus.cc:2258] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:22.397645 12990 tablet_replica.cc:330] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: stopping tablet replica
I20260928 16:48:22.398108 12990 raft_consensus.cc:2229] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:22.398682 12990 raft_consensus.cc:2258] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:22.401672 12990 ts_tablet_manager.cc:1905] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:22.401772 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.364054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.26.101.1:45405)
0928 16:48:22.364253 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.364260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.387423 (+ 23163us) server_negotiation.cc:407] Connection header received
0928 16:48:22.388004 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.388010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.388135 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.388225 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:22.388992 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.390238 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.392582 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.393310 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.393417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.399683 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:22.399698 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:22.399720 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:22.400617 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:22.400698 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:22.400733 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:22.400889 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.401164 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.401331 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:48:22.403519 12679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:22.404891 12989 ts_tablet_manager.cc:1905] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:22.413978 12990 ts_tablet_manager.cc:1918] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:22.414184 12990 log.cc:1196] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/27bc9f513dfe48f6b738bee09d14f411
I20260928 16:48:22.414762 12990 ts_tablet_manager.cc:1939] T 27bc9f513dfe48f6b738bee09d14f411 P 79006f8409014e7d965fca54a7b1627e: Deleting consensus metadata
I20260928 16:48:22.415683 12989 ts_tablet_manager.cc:1918] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:22.415939 12989 log.cc:1196] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/27bc9f513dfe48f6b738bee09d14f411
I20260928 16:48:22.416553 12989 ts_tablet_manager.cc:1939] T 27bc9f513dfe48f6b738bee09d14f411 P a6c4490dc2694d4baaecb69467509194: Deleting consensus metadata
I20260928 16:48:22.417040 12483 catalog_manager.cc:4841] TS 79006f8409014e7d965fca54a7b1627e (127.26.101.2:33755): tablet 27bc9f513dfe48f6b738bee09d14f411 (table test-table [id=2e00dec24ef14c43b01615a95b690222]) successfully deleted
I20260928 16:48:22.418361 12480 catalog_manager.cc:4841] TS a6c4490dc2694d4baaecb69467509194 (127.26.101.3:39773): tablet 27bc9f513dfe48f6b738bee09d14f411 (table test-table [id=2e00dec24ef14c43b01615a95b690222]) successfully deleted
I20260928 16:48:22.508098 12988 tablet_replica.cc:330] T 27bc9f513dfe48f6b738bee09d14f411 P eae8a616d8c642de8fbddb5f58c6eec9: stopping tablet replica
I20260928 16:48:22.508286 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12556
W20260928 16:48:22.524783 12482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.101.1:45405 (error 108)
I20260928 16:48:22.524994 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12688
I20260928 16:48:22.526978 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.526213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47428)
0928 16:48:22.526615 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.526751 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":216,"threads_started":1}
W20260928 16:48:22.527583 12482 catalog_manager.cc:4584] TS eae8a616d8c642de8fbddb5f58c6eec9 (127.26.101.1:45405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27bc9f513dfe48f6b738bee09d14f411: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
I20260928 16:48:22.533440 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.525924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.26.101.2:33755)
0928 16:48:22.526148 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.526154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.526201 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:48:22.526274 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.526280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.526404 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.526509 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:22.527136 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.528328 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.529667 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.530341 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.530483 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.531651 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:22.531664 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:22.531685 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:22.532603 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:22.532672 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:22.532699 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:22.532822 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.532927 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.533052 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:48:22.534279 12810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:22.581278 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.580673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47444)
0928 16:48:22.580985 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.581098 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260928 16:48:22.582460 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.582025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47448)
0928 16:48:22.582202 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.582294 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:48:22.624248 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12688
I20260928 16:48:22.640105 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12818
I20260928 16:48:22.643234 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.642699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47454)
0928 16:48:22.642926 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.643027 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 16:48:22.644205 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.643865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47466)
0928 16:48:22.644004 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.644080 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:48:22.648094 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.641090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.26.101.3:39773)
0928 16:48:22.641283 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:22.641287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:22.641308 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:22.641495 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:22.641499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:22.641583 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:22.641667 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:22.642151 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.643045 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.644925 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:22.645365 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:22.645472 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:22.646661 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:22.646669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:22.646687 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:22.647326 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:22.647379 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:22.647402 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:22.647481 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:22.647673 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:48:22.647755 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:48:22.648811 12942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:22.725131 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.724520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47468)
0928 16:48:22.724807 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.724943 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 16:48:22.726316 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:22.725830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:45405 (local address 127.0.0.1:47482)
0928 16:48:22.726049 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:22.726139 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 16:48:22.768683 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12818
I20260928 16:48:22.785987 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 12465
I20260928 16:48:22.787096 12527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:22.898253 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 12465
2026-09-28T16:48:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2968 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:23 dist-test-slave-t0tw krb5kdc[13001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:23 dist-test-slave-t0tw krb5kdc[13001](info): set up 2 sockets
Sep 28 16:48:23 dist-test-slave-t0tw krb5kdc[13001](info): commencing operation
krb5kdc: starting...
W20260928 16:48:23.930727 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:24 dist-test-slave-t0tw krb5kdc[13001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:48:24.191288 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45581
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:24.525552 13014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:24.525857 13014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:24.525951 13014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:24.533210 13014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:24.533337 13014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:24.533398 13014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:24.533453 13014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:24.533501 13014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:24.543615 13014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:24.544976 13014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:24.546268 13014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:24.557806 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:24.557849 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:24.557889 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:24.559181 13014 server_base.cc:1034] running on GCE node
I20260928 16:48:24.560001 13014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:24.561486 13014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:24.562698 13014 hybrid_clock.cc:648] HybridClock initialized: now 1790614104562649 us; error 46 us; skew 500 ppm
I20260928 16:48:24.563053 13014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:24 dist-test-slave-t0tw krb5kdc[13001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:24.566205 13014 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:48:24.567744 13014 webserver.cc:460] Webserver started at http://127.26.101.62:44537/ using document root <none> and password file <none>
I20260928 16:48:24.568441 13014 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:24.568564 13014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:24.568851 13014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:24.571943 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "4fa15f2af73e4138b5a219a8754a8d57"
format_stamp: "Formatted at 2026-09-28 16:48:24 on dist-test-slave-t0tw"
server_key: "08c7c4a34b6e3ce3fff9ad0d58a05414"
server_key_iv: "9ebf39d0163f8483a540421dab8388d7"
server_key_version: "encryptionkey@0"
I20260928 16:48:24.572690 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "4fa15f2af73e4138b5a219a8754a8d57"
format_stamp: "Formatted at 2026-09-28 16:48:24 on dist-test-slave-t0tw"
server_key: "08c7c4a34b6e3ce3fff9ad0d58a05414"
server_key_iv: "9ebf39d0163f8483a540421dab8388d7"
server_key_version: "encryptionkey@0"
I20260928 16:48:24.578023 13014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 16:48:24.581555 13029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:24.582953 13014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:48:24.583190 13014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "4fa15f2af73e4138b5a219a8754a8d57"
format_stamp: "Formatted at 2026-09-28 16:48:24 on dist-test-slave-t0tw"
server_key: "08c7c4a34b6e3ce3fff9ad0d58a05414"
server_key_iv: "9ebf39d0163f8483a540421dab8388d7"
server_key_version: "encryptionkey@0"
I20260928 16:48:24.583387 13014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:24.600936 13014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:24.605626 13014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:24.605911 13014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:24.623852 13014 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45581
I20260928 16:48:24.623872 13080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45581 every 8 connection(s)
I20260928 16:48:24.625205 13014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:24.625910 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13014
I20260928 16:48:24.626093 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:24.626652 27028 external_mini_cluster.cc:1428] Setting key 22edee89614416c9d5d38727728a7e3e
I20260928 16:48:24.629601 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:48:24 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:24.642818 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: Bootstrap starting.
I20260928 16:48:24.647159 13081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:24.648370 13081 log.cc:824] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:24.651737 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:24.628505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.26.101.62:45581)
0928 16:48:24.629286 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:24.629299 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:24.629344 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:48:24.631024 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:24.631049 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:24.631490 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:24.631824 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:24.633780 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:24.635231 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:24.637173 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:24.637888 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:24.638021 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:24.642911 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:24.642924 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:24.642944 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:24.642997 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:24.646848 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:24.647723 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:24.647731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:24.647739 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:24.647956 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:24.648319 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:24.648324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:24.648327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:24.649114 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:24.649381 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:24.649944 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 16:48:24.650404 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":345,"threads_started":1}
I20260928 16:48:24.651995 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: No bootstrap required, opened a new log
I20260928 16:48:24.659864 13081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } }
I20260928 16:48:24.660243 13081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:24.660344 13081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa15f2af73e4138b5a219a8754a8d57, State: Initialized, Role: FOLLOWER
I20260928 16:48:24.660962 13081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } }
I20260928 16:48:24.661191 13081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:24.661309 13081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:24.661475 13081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:24.663679 13081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } }
I20260928 16:48:24.664178 13081 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa15f2af73e4138b5a219a8754a8d57; no voters:
I20260928 16:48:24.664762 13081 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:24.664951 13086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:24.665927 13086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa15f2af73e4138b5a219a8754a8d57, State: Running, Role: LEADER
I20260928 16:48:24.666488 13086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } }
I20260928 16:48:24.666986 13081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:24.669237 13087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa15f2af73e4138b5a219a8754a8d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } } }
I20260928 16:48:24.669349 13088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa15f2af73e4138b5a219a8754a8d57. Latest consensus state: current_term: 1 leader_uuid: "4fa15f2af73e4138b5a219a8754a8d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa15f2af73e4138b5a219a8754a8d57" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45581 } } }
I20260928 16:48:24.669714 13087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:24.669720 13088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:24.670639 13093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:24.676268 13093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:24.697034 13093 catalog_manager.cc:1347] Generated new cluster ID: ffd8931012b543be9485c93263212132
I20260928 16:48:24.697178 13093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:24.719686 13093 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:24.721025 13093 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:24.736975 13093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: Generated new TSK 0
I20260928 16:48:24.737901 13093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:48:24.793959 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:25.131669 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:25.131970 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:25.132090 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:25.139428 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:25.139544 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:25.139694 13107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:25.150676 13107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:25.152056 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:25.153383 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:25.166352 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:25.166378 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:25.166327 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:25.167191 13107 server_base.cc:1034] running on GCE node
I20260928 16:48:25.167862 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:25.168921 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:25.170147 13107 hybrid_clock.cc:648] HybridClock initialized: now 1790614105170084 us; error 46 us; skew 500 ppm
I20260928 16:48:25.170516 13107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:25 dist-test-slave-t0tw krb5kdc[13001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614105, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:25.173668 13107 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:48:25.175117 13107 webserver.cc:460] Webserver started at http://127.26.101.1:44701/ using document root <none> and password file <none>
I20260928 16:48:25.175820 13107 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:25.175942 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:25.176244 13107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:25.179211 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "c8349efa3b2c48808457a9f9bc2dd77c"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "dfb3bca6ab7b0251b9f492e90bcbe363"
server_key_iv: "bb75e9481c76429ba62ec7793b65d665"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.179991 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "c8349efa3b2c48808457a9f9bc2dd77c"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "dfb3bca6ab7b0251b9f492e90bcbe363"
server_key_iv: "bb75e9481c76429ba62ec7793b65d665"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.185304 13107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:25.189209 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.190613 13107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:25.190784 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "c8349efa3b2c48808457a9f9bc2dd77c"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "dfb3bca6ab7b0251b9f492e90bcbe363"
server_key_iv: "bb75e9481c76429ba62ec7793b65d665"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.190989 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:25.218104 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:25.222640 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:25.222929 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:25.223968 13107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:25.225364 13107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:25.225447 13107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.225574 13107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:25.225629 13107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.256763 13107 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41277
I20260928 16:48:25.256819 13235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41277 every 8 connection(s)
I20260928 16:48:25.258047 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:25.259647 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13107
I20260928 16:48:25.259809 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:25.260242 27028 external_mini_cluster.cc:1428] Setting key f599968c8151287b93deb8c321e1c949
Sep 28 16:48:25 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614105, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:25.281296 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.260836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:52447 (local address 127.26.101.62:45581)
0928 16:48:25.261938 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:25.261945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:25.262392 (+ 447us) server_negotiation.cc:407] Connection header received
0928 16:48:25.265292 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:25.265297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:25.265386 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:25.265480 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:25.267654 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.268498 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.270626 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.271017 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.271150 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:25.275287 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:25.275292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:25.275295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:25.275333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:25.277890 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:25.278753 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:25.278758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:25.278761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:25.278902 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:25.279430 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:25.279434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:25.279436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:25.279728 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:25.279879 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:25.280655 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 16:48:25.280925 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":239,"threads_started":1}
I20260928 16:48:25.282416 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.261402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45581 (local address 127.26.101.1:52447)
0928 16:48:25.262204 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:25.262255 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:25.264893 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:25.265779 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:25.265790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:25.266537 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:25.267463 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:25.267493 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.268666 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.268672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:25.270487 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:25.270511 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.271272 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.271279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:25.271556 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:25.272363 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:25.272391 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:25.275001 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:25.278062 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:25.278073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:25.278092 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:25.278610 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:25.279018 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:25.279023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:25.279027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:25.279304 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:25.279861 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:25.279870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:25.280247 (+ 377us) client_negotiation.cc:769] Sending connection context
0928 16:48:25.280509 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 16:48:25.281085 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260928 16:48:25.284461 13236 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45581
I20260928 16:48:25.284811 13236 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:25.285568 13236 heartbeater.cc:511] Master 127.26.101.62:45581 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
I20260928 16:48:25.288451 13046 ts_manager.cc:194] Registered new tserver with Master: c8349efa3b2c48808457a9f9bc2dd77c (127.26.101.1:41277)
I20260928 16:48:25.290068 13046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:52447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:48:25.320323 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:25.647612 13242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:25.647918 13242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:25.648061 13242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:25.655478 13242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:25.655610 13242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:25.655771 13242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:25.665863 13242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:25.667301 13242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:25.668659 13242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:25.681815 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:25.681789 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:25.682060 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:25.682565 13242 server_base.cc:1034] running on GCE node
I20260928 16:48:25.683332 13242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:25.684437 13242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:25.685650 13242 hybrid_clock.cc:648] HybridClock initialized: now 1790614105685601 us; error 48 us; skew 500 ppm
I20260928 16:48:25.685974 13242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:25 dist-test-slave-t0tw krb5kdc[13001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614105, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:25.689379 13242 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:48:25.690943 13242 webserver.cc:460] Webserver started at http://127.26.101.2:38207/ using document root <none> and password file <none>
I20260928 16:48:25.691866 13242 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:25.692009 13242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:25.692342 13242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:25.695591 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "549b875dc00e4ffcbff9da8d75e1f6ef"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "2160a5604d907484216cda4b4337d6e2"
server_key_iv: "3781de290c6e0f58a24bfe0bc4c821c5"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.696415 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "549b875dc00e4ffcbff9da8d75e1f6ef"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "2160a5604d907484216cda4b4337d6e2"
server_key_iv: "3781de290c6e0f58a24bfe0bc4c821c5"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.702024 13242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:25.706004 13258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.707662 13242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:25.707856 13242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "549b875dc00e4ffcbff9da8d75e1f6ef"
format_stamp: "Formatted at 2026-09-28 16:48:25 on dist-test-slave-t0tw"
server_key: "2160a5604d907484216cda4b4337d6e2"
server_key_iv: "3781de290c6e0f58a24bfe0bc4c821c5"
server_key_version: "encryptionkey@0"
I20260928 16:48:25.708074 13242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:25.739006 13242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:25.743628 13242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:25.743922 13242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:25.744926 13242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:25.746296 13242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:25.746385 13242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.746505 13242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:25.746567 13242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:25.777513 13242 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:41783
I20260928 16:48:25.777563 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:41783 every 8 connection(s)
I20260928 16:48:25.778833 13242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:25.785744 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13242
I20260928 16:48:25.785956 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:25.786455 27028 external_mini_cluster.cc:1428] Setting key 0b4a8f4a67ba5eae0b46f061691dfcc8
Sep 28 16:48:25 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614105, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:25.799304 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.781385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:49237 (local address 127.26.101.62:45581)
0928 16:48:25.781559 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:25.781565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:25.782773 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 16:48:25.784791 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:25.784797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:25.784899 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:25.784989 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:25.786674 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.787660 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.789392 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.789834 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.789953 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:25.793439 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:25.793444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:25.793448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:25.793482 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:25.796148 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:25.796879 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:25.796883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:25.796886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:25.797022 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:25.797506 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:25.797509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:25.797511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:25.797802 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:25.797922 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:25.798787 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 16:48:25.798978 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:48:25.800581 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:25.781846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45581 (local address 127.26.101.2:49237)
0928 16:48:25.782600 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:25.782640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:25.784474 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:25.785255 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:25.785265 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:25.785822 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:25.786464 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:25.786485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.787816 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.787820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:25.789242 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:25.789257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:25.790070 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:25.790078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:25.790312 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:25.790966 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:25.790990 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:25.793189 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:25.796298 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:25.796308 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:25.796325 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:25.796757 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:25.797135 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:25.797141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:25.797145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:25.797402 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:25.797918 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:25.797927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:25.798373 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 16:48:25.798678 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 16:48:25.799248 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260928 16:48:25.802090 13372 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45581
I20260928 16:48:25.802434 13372 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:25.803318 13372 heartbeater.cc:511] Master 127.26.101.62:45581 requested a full tablet report, sending...
I20260928 16:48:25.804996 13046 ts_manager.cc:194] Registered new tserver with Master: 549b875dc00e4ffcbff9da8d75e1f6ef (127.26.101.2:41783)
I20260928 16:48:25.805814 13046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:49237
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:48:25.843307 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:26.173149 13377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:26.173446 13377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:26.173583 13377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:26.180778 13377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:26.180914 13377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:26.181064 13377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:26.191510 13377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:45581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:26.192830 13377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:26.194120 13377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:26.207229 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:26.207254 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:26.207597 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:26.208016 13377 server_base.cc:1034] running on GCE node
I20260928 16:48:26.208652 13377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:26.209683 13377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:26.210897 13377 hybrid_clock.cc:648] HybridClock initialized: now 1790614106210837 us; error 50 us; skew 500 ppm
I20260928 16:48:26.211272 13377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:26 dist-test-slave-t0tw krb5kdc[13001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614106, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:26.214454 13377 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:48:26.216008 13377 webserver.cc:460] Webserver started at http://127.26.101.3:45765/ using document root <none> and password file <none>
I20260928 16:48:26.216687 13377 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:26.216809 13377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:26.217116 13377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:26.220124 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "3a8d44226f95493f8431befc7e5d4227"
format_stamp: "Formatted at 2026-09-28 16:48:26 on dist-test-slave-t0tw"
server_key: "cdc1085371af8ec7b38d807b05f5fb22"
server_key_iv: "00a87388fcd4e03b369ed4138c7fc368"
server_key_version: "encryptionkey@0"
I20260928 16:48:26.220866 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "3a8d44226f95493f8431befc7e5d4227"
format_stamp: "Formatted at 2026-09-28 16:48:26 on dist-test-slave-t0tw"
server_key: "cdc1085371af8ec7b38d807b05f5fb22"
server_key_iv: "00a87388fcd4e03b369ed4138c7fc368"
server_key_version: "encryptionkey@0"
I20260928 16:48:26.226356 13377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:26.229841 13392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:26.231318 13377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:26.231480 13377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "3a8d44226f95493f8431befc7e5d4227"
format_stamp: "Formatted at 2026-09-28 16:48:26 on dist-test-slave-t0tw"
server_key: "cdc1085371af8ec7b38d807b05f5fb22"
server_key_iv: "00a87388fcd4e03b369ed4138c7fc368"
server_key_version: "encryptionkey@0"
I20260928 16:48:26.231673 13377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:26.246407 13377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:26.250381 13377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:26.250656 13377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:26.251706 13377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:26.253068 13377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:26.253147 13377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:26.253284 13377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:26.253342 13377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:26.284492 13377 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:40909
I20260928 16:48:26.284533 13507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:40909 every 8 connection(s)
I20260928 16:48:26.285753 13377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:26.287787 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13377
I20260928 16:48:26.287969 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:26.288445 27028 external_mini_cluster.cc:1428] Setting key e7eb22795b85a4ed99a7aa512fdfd108
I20260928 16:48:26.293481 13236 heartbeater.cc:503] Master 127.26.101.62:45581 was elected leader, sending a full tablet report...
Sep 28 16:48:26 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614106, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:26.306437 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.288473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:55545 (local address 127.26.101.62:45581)
0928 16:48:26.288672 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.288678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.289896 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 16:48:26.291998 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.292002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.292072 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.292156 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:26.293665 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.294680 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.296406 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.296807 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.296901 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.300303 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:26.300308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:26.300311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:26.300344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:26.303558 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.304368 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.304372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.304375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.304508 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.304869 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.304871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.304873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.305145 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:26.305254 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.305960 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.306144 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 16:48:26.307390 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.288975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:45581 (local address 127.26.101.3:55545)
0928 16:48:26.289734 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.289788 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.291742 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.292380 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.292388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.292887 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:26.293476 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.293495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.294830 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.294836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.296263 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.296277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.296982 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.296986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.297186 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.297779 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:26.297801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:26.300072 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:26.303714 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:26.303724 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:26.303742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:26.304228 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:26.304589 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:26.304592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:26.304594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:26.304787 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:26.305235 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:26.305241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:26.305624 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.305845 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.306275 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":304,"threads_started":1}
I20260928 16:48:26.308645 13508 heartbeater.cc:347] Connected to a master server at 127.26.101.62:45581
I20260928 16:48:26.308947 13508 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:26.309644 13508 heartbeater.cc:511] Master 127.26.101.62:45581 requested a full tablet report, sending...
I20260928 16:48:26.311017 13046 ts_manager.cc:194] Registered new tserver with Master: 3a8d44226f95493f8431befc7e5d4227 (127.26.101.3:40909)
I20260928 16:48:26.311861 13046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:55545
I20260928 16:48:26.316115 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:26.333287 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.320211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.26.101.62:45581)
0928 16:48:26.320448 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.320456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.320730 (+ 274us) server_negotiation.cc:407] Connection header received
0928 16:48:26.320918 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.320925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.320993 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.321075 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:26.322440 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.323238 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.324610 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.325060 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.325175 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.327931 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:26.327938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:26.327941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:26.327975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:26.330321 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.331197 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.331203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.331207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.331355 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.331804 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.331809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.331812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.332134 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:26.332408 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.332800 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.332987 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 16:48:26.337163 13046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:26.343926 13046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:26.380760 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.355844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40909 (local address 127.0.0.1:60958)
0928 16:48:26.371507 (+ 15663us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.371574 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.371818 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.372771 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.372780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.372818 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:26.373432 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.373448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.375625 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.375630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.378197 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.378216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.379716 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.379723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.379935 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.380007 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.380138 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.380333 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15340,"mutex_wait_us":79,"thread_start_us":16163,"threads_started":1}
I20260928 16:48:26.381479 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.357089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.26.101.1:41277)
0928 16:48:26.372962 (+ 15873us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.372970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.373711 (+ 741us) server_negotiation.cc:407] Connection header received
0928 16:48:26.373832 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.373840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.374088 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.374257 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:26.375189 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.376315 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.378683 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.380010 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.380154 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.380383 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.380965 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.381141 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15700,"thread_start_us":15552,"threads_started":1}
I20260928 16:48:26.381975 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.355979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.26.101.3:40909)
0928 16:48:26.372070 (+ 16091us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.372080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.372118 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:48:26.372229 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.372236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.372502 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.373674 (+ 1172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:26.373965 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.375487 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.378370 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.379536 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.379662 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.379896 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.381482 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.381708 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15924,"thread_start_us":15772,"threads_started":1}
I20260928 16:48:26.382032 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.357569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.26.101.2:41783)
0928 16:48:26.372477 (+ 14908us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.372485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.374034 (+ 1549us) server_negotiation.cc:407] Connection header received
0928 16:48:26.374147 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.374154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.374375 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.374549 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:26.375367 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.376860 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.379336 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.380662 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.380784 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.380970 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.381569 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.381730 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14758,"thread_start_us":14616,"threads_started":1}
I20260928 16:48:26.382507 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.356092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38484)
0928 16:48:26.373404 (+ 17312us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.373423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.373558 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.374509 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.374514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.374538 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:26.374967 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.374980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.376462 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.376467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.378519 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.378534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.380231 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.380238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.380411 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.380444 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.380561 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.380695 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17176,"mutex_wait_us":66,"thread_start_us":17411,"threads_started":1}
I20260928 16:48:26.382711 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.356417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:41783 (local address 127.0.0.1:53574)
0928 16:48:26.373781 (+ 17364us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.373797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.373909 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.374780 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.374785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.374806 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:26.375218 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.375230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.377031 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.377037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.379160 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.379175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.381117 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.381123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.381282 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.381315 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.381394 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.382358 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17226,"mutex_wait_us":27,"thread_start_us":16715,"threads_started":1}
I20260928 16:48:26.385638 13441 tablet_service.cc:1467] Processing CreateTablet for tablet 90276962ac8f471aad219b69e9c3c83b (DEFAULT_TABLE table=test-table [id=49665d9ef7b340e48997d57c88abd428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:26.385841 13171 tablet_service.cc:1467] Processing CreateTablet for tablet 90276962ac8f471aad219b69e9c3c83b (DEFAULT_TABLE table=test-table [id=49665d9ef7b340e48997d57c88abd428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:26.387202 13441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90276962ac8f471aad219b69e9c3c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:26.387327 13171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90276962ac8f471aad219b69e9c3c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:26.387813 13306 tablet_service.cc:1467] Processing CreateTablet for tablet 90276962ac8f471aad219b69e9c3c83b (DEFAULT_TABLE table=test-table [id=49665d9ef7b340e48997d57c88abd428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:26.389133 13306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90276962ac8f471aad219b69e9c3c83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:26.399276 13526 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Bootstrap starting.
I20260928 16:48:26.402468 13527 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Bootstrap starting.
I20260928 16:48:26.402797 13526 tablet_bootstrap.cc:654] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:26.404281 13526 log.cc:824] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:26.406209 13527 tablet_bootstrap.cc:654] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:26.407606 13527 log.cc:824] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:26.407927 13528 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Bootstrap starting.
I20260928 16:48:26.411024 13528 tablet_bootstrap.cc:654] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:26.411537 13527 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: No bootstrap required, opened a new log
I20260928 16:48:26.411542 13526 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: No bootstrap required, opened a new log
I20260928 16:48:26.411815 13527 ts_tablet_manager.cc:1397] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 16:48:26.411815 13526 ts_tablet_manager.cc:1397] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260928 16:48:26.412128 13528 log.cc:824] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:26.415460 13528 tablet_bootstrap.cc:492] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: No bootstrap required, opened a new log
I20260928 16:48:26.415705 13528 ts_tablet_manager.cc:1397] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260928 16:48:26.422366 13526 raft_consensus.cc:348] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.422377 13527 raft_consensus.cc:348] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.422879 13526 raft_consensus.cc:374] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:26.422871 13527 raft_consensus.cc:374] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:26.422698 13528 raft_consensus.cc:348] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.422976 13527 raft_consensus.cc:729] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8349efa3b2c48808457a9f9bc2dd77c, State: Initialized, Role: FOLLOWER
I20260928 16:48:26.422976 13526 raft_consensus.cc:729] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8d44226f95493f8431befc7e5d4227, State: Initialized, Role: FOLLOWER
I20260928 16:48:26.423121 13528 raft_consensus.cc:374] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:26.423220 13528 raft_consensus.cc:729] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549b875dc00e4ffcbff9da8d75e1f6ef, State: Initialized, Role: FOLLOWER
I20260928 16:48:26.423651 13527 consensus_queue.cc:260] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.423650 13526 consensus_queue.cc:260] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.423858 13528 consensus_queue.cc:260] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.425588 13527 ts_tablet_manager.cc:1428] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 16:48:26.426294 13508 heartbeater.cc:503] Master 127.26.101.62:45581 was elected leader, sending a full tablet report...
I20260928 16:48:26.427738 13526 ts_tablet_manager.cc:1428] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 16:48:26.431154 13372 heartbeater.cc:503] Master 127.26.101.62:45581 was elected leader, sending a full tablet report...
I20260928 16:48:26.431166 13528 ts_tablet_manager.cc:1428] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 16:48:26.449126 13534 raft_consensus.cc:482] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:26.449445 13534 raft_consensus.cc:504] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.451112 13534 leader_election.cc:288] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8d44226f95493f8431befc7e5d4227 (127.26.101.3:40909), c8349efa3b2c48808457a9f9bc2dd77c (127.26.101.1:41277)
I20260928 16:48:26.458020 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.451640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:55183 (local address 127.26.101.3:40909)
0928 16:48:26.451865 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.451873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.452567 (+ 694us) server_negotiation.cc:407] Connection header received
0928 16:48:26.452747 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.452751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.452815 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.452890 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:26.453451 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.454224 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.456160 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.457090 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.457215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.457269 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.457631 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.457761 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:48:26.458361 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.451601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:40909 (local address 127.26.101.2:55183)
0928 16:48:26.452447 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.452464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.452602 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.452926 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.452931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.452955 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:26.453306 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.453320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.454377 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.454382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.456005 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.456018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.457313 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.457320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.457483 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.457519 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.457608 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.457754 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":1,"thread_start_us":481,"threads_started":1}
I20260928 16:48:26.458503 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.451590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:51401 (local address 127.26.101.1:41277)
0928 16:48:26.451919 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.451925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.452636 (+ 711us) server_negotiation.cc:407] Connection header received
0928 16:48:26.452828 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.452833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.452895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.452969 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:26.453566 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.454416 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.456587 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.457489 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.457657 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.457724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.458124 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.458272 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260928 16:48:26.458684 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.451713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.26.101.2:51401)
0928 16:48:26.452534 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.452553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:26.452697 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:26.453002 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:26.453011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:26.453052 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:26.453424 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.453433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.454582 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.454587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.456420 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:26.456435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.457720 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.457727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:26.457941 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.457977 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:48:26.458082 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:48:26.458224 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":20,"thread_start_us":448,"threads_started":1}
I20260928 16:48:26.459547 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90276962ac8f471aad219b69e9c3c83b" candidate_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8d44226f95493f8431befc7e5d4227" is_pre_election: true
I20260928 16:48:26.459719 13191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90276962ac8f471aad219b69e9c3c83b" candidate_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" is_pre_election: true
I20260928 16:48:26.460071 13462 raft_consensus.cc:2454] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549b875dc00e4ffcbff9da8d75e1f6ef in term 0.
I20260928 16:48:26.460234 13191 raft_consensus.cc:2454] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549b875dc00e4ffcbff9da8d75e1f6ef in term 0.
I20260928 16:48:26.460743 13262 leader_election.cc:302] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8d44226f95493f8431befc7e5d4227, 549b875dc00e4ffcbff9da8d75e1f6ef; no voters:
I20260928 16:48:26.461194 13534 raft_consensus.cc:2781] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:26.461318 13534 raft_consensus.cc:482] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:26.461418 13534 raft_consensus.cc:3037] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:26.463677 13534 raft_consensus.cc:504] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.464275 13534 leader_election.cc:288] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [CANDIDATE]: Term 1 election: Requested vote from peers 3a8d44226f95493f8431befc7e5d4227 (127.26.101.3:40909), c8349efa3b2c48808457a9f9bc2dd77c (127.26.101.1:41277)
I20260928 16:48:26.464782 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90276962ac8f471aad219b69e9c3c83b" candidate_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8d44226f95493f8431befc7e5d4227"
I20260928 16:48:26.464804 13191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90276962ac8f471aad219b69e9c3c83b" candidate_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8349efa3b2c48808457a9f9bc2dd77c"
I20260928 16:48:26.464995 13462 raft_consensus.cc:3037] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:26.465001 13191 raft_consensus.cc:3037] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:26.467308 13191 raft_consensus.cc:2454] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549b875dc00e4ffcbff9da8d75e1f6ef in term 1.
I20260928 16:48:26.467338 13462 raft_consensus.cc:2454] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549b875dc00e4ffcbff9da8d75e1f6ef in term 1.
I20260928 16:48:26.467814 13260 leader_election.cc:302] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a8d44226f95493f8431befc7e5d4227, 549b875dc00e4ffcbff9da8d75e1f6ef, c8349efa3b2c48808457a9f9bc2dd77c; no voters:
I20260928 16:48:26.468178 13534 raft_consensus.cc:2781] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:26.468619 13534 raft_consensus.cc:686] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 1 LEADER]: Becoming Leader. State: Replica: 549b875dc00e4ffcbff9da8d75e1f6ef, State: Running, Role: LEADER
I20260928 16:48:26.469081 13534 consensus_queue.cc:237] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } }
I20260928 16:48:26.474012 13045 catalog_manager.cc:5495] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 549b875dc00e4ffcbff9da8d75e1f6ef (127.26.101.2). New cstate: current_term: 1 leader_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "549b875dc00e4ffcbff9da8d75e1f6ef" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41783 } health_report { overall_health: HEALTHY } } }
W20260928 16:48:26.510684 13237 tablet.cc:2369] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:26.512818 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.506567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.26.101.2:41783)
0928 16:48:26.506778 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.506785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.506806 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:26.507028 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.507033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.507138 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.507281 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:26.507678 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.508676 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.510215 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.510849 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.510984 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.511704 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:26.512286 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:26.512425 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.512565 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.512607 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:48:26.520818 13191 raft_consensus.cc:1261] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 1 FOLLOWER]: Refusing update from remote peer 549b875dc00e4ffcbff9da8d75e1f6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:26.520819 13462 raft_consensus.cc:1261] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 1 FOLLOWER]: Refusing update from remote peer 549b875dc00e4ffcbff9da8d75e1f6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:26.521802 13537 consensus_queue.cc:1035] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8349efa3b2c48808457a9f9bc2dd77c" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:26.522185 13534 consensus_queue.cc:1035] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8d44226f95493f8431befc7e5d4227" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:48:26.530872 13373 tablet.cc:2369] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:26.543797 13543 mvcc.cc:205] Tried to move back new op lower bound from 7334355380299567104 to 7334355380101279744. Current Snapshot: MvccSnapshot[applied={T|T < 7334355380299567104 or (T in {7334355380299567104})}]
I20260928 16:48:26.552054 13045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:26.552392 13045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:26.556289 13045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fa15f2af73e4138b5a219a8754a8d57: Sending DeleteTablet for 3 replicas of tablet 90276962ac8f471aad219b69e9c3c83b
I20260928 16:48:26.557540 13171 tablet_service.cc:1514] Processing DeleteTablet for tablet 90276962ac8f471aad219b69e9c3c83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:26 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:38484
I20260928 16:48:26.557965 13441 tablet_service.cc:1514] Processing DeleteTablet for tablet 90276962ac8f471aad219b69e9c3c83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:26 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:60958
I20260928 16:48:26.559227 13306 tablet_service.cc:1514] Processing DeleteTablet for tablet 90276962ac8f471aad219b69e9c3c83b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:26 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:53574
I20260928 16:48:26.563212 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13107
I20260928 16:48:26.583614 13552 tablet_replica.cc:330] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: stopping tablet replica
I20260928 16:48:26.582944 13553 tablet_replica.cc:330] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: stopping tablet replica
I20260928 16:48:26.584379 13553 raft_consensus.cc:2229] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:26.584975 13552 raft_consensus.cc:2229] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:26.585768 13553 raft_consensus.cc:2258] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:26.586354 13552 raft_consensus.cc:2258] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 16:48:26 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:48:26.589505 13553 ts_tablet_manager.cc:1905] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:26.590965 13552 ts_tablet_manager.cc:1905] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:26.599468 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.564995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.26.101.1:41277)
0928 16:48:26.565198 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.565204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.575795 (+ 10591us) server_negotiation.cc:407] Connection header received
0928 16:48:26.576347 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.576356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.576456 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.576534 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:26.578545 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.579964 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.582160 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.582911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.583021 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.590668 (+ 7647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:26.590695 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:26.590716 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:26.590793 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:26.595290 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.596355 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.596367 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.596371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.596594 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.597102 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.597108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.597112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.597654 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:26.597968 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.598644 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.598940 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:26.599886 13551 tablet_replica.cc:330] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c: stopping tablet replica
I20260928 16:48:26.600368 13551 raft_consensus.cc:2229] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:26.600832 13231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:26.604818 13552 ts_tablet_manager.cc:1918] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:26.605016 13552 log.cc:1196] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/90276962ac8f471aad219b69e9c3c83b
I20260928 16:48:26.605551 13552 ts_tablet_manager.cc:1939] T 90276962ac8f471aad219b69e9c3c83b P 3a8d44226f95493f8431befc7e5d4227: Deleting consensus metadata
I20260928 16:48:26.607688 13033 catalog_manager.cc:4841] TS 3a8d44226f95493f8431befc7e5d4227 (127.26.101.3:40909): tablet 90276962ac8f471aad219b69e9c3c83b (table test-table [id=49665d9ef7b340e48997d57c88abd428]) successfully deleted
I20260928 16:48:26.610981 13553 ts_tablet_manager.cc:1918] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:26.611189 13553 log.cc:1196] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/90276962ac8f471aad219b69e9c3c83b
I20260928 16:48:26.611770 13553 ts_tablet_manager.cc:1939] T 90276962ac8f471aad219b69e9c3c83b P 549b875dc00e4ffcbff9da8d75e1f6ef: Deleting consensus metadata
I20260928 16:48:26.613792 13032 catalog_manager.cc:4841] TS 549b875dc00e4ffcbff9da8d75e1f6ef (127.26.101.2:41783): tablet 90276962ac8f471aad219b69e9c3c83b (table test-table [id=49665d9ef7b340e48997d57c88abd428]) successfully deleted
I20260928 16:48:26.747629 13551 raft_consensus.cc:2258] T 90276962ac8f471aad219b69e9c3c83b P c8349efa3b2c48808457a9f9bc2dd77c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:26.748281 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13107
I20260928 16:48:26.763934 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13242
W20260928 16:48:26.764058 13031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.101.1:41277 (error 108)
I20260928 16:48:26.766219 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.765807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38506)
0928 16:48:26.765977 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.766075 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 16:48:26.766523 13031 catalog_manager.cc:4584] TS c8349efa3b2c48808457a9f9bc2dd77c (127.26.101.1:41277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90276962ac8f471aad219b69e9c3c83b: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Sep 28 16:48:26 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:48:26.782519 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.765102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.26.101.2:41783)
0928 16:48:26.765302 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.765308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.765329 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:26.765383 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.765387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.765451 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.765535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:26.767248 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.768133 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.769968 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.770437 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.770531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.775806 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:26.775820 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:26.775835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:26.775877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:26.779345 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.780083 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.780090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.780092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.780264 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.780632 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.780636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.780638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.781016 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:26.781374 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.781768 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.782113 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:48:26.783277 13367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:26.820204 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.819711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38514)
0928 16:48:26.819930 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.820034 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 16:48:26.821424 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.821030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38524)
0928 16:48:26.821191 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.821268 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:48:26.882012 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.881528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38532)
0928 16:48:26.881735 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.881842 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 16:48:26.883124 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.882704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38544)
0928 16:48:26.882846 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.882937 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 16:48:26.898034 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13242
I20260928 16:48:26.916533 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13377
Sep 28 16:48:26 dist-test-slave-t0tw krb5kdc[13001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:48:26.932164 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.917382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.26.101.3:40909)
0928 16:48:26.917624 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:26.917628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:26.917649 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:48:26.917821 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:26.917825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:26.917892 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:26.917981 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:26.919387 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.920146 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.921385 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:26.921798 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:26.921890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:26.926375 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:26.926390 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:26.926407 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:26.926455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:26.929208 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.929987 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.929996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.929998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.930144 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:26.930509 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:26.930512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:26.930514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:26.930891 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:26.931167 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:26.931594 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 16:48:26.931824 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 16:48:26.932852 13503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:26.963967 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.963338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38554)
0928 16:48:26.963595 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.963715 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 16:48:26.965164 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:26.964673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38570)
0928 16:48:26.964914 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:26.965010 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 16:48:27.021978 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13377
I20260928 16:48:27.036834 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13014
I20260928 16:48:27.037947 13076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:27.150722 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:27.150119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41277 (local address 127.0.0.1:38582)
0928 16:48:27.150289 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:27.150377 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 16:48:27.150995 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13014
2026-09-28T16:48:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4255 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T16:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:27Z Disabled control of system clock
I20260928 16:48:27.228209 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40669
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:40669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:27.560040 13563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:27.560357 13563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:27.560451 13563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:27.567840 13563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:27.567965 13563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:27.568001 13563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:27.568058 13563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:27.568100 13563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:27.577499 13563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:40669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:40669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:27.578840 13563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:27.580647 13563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:27.592132 13568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:27.592175 13571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:27.592211 13569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:27.593204 13563 server_base.cc:1034] running on GCE node
I20260928 16:48:27.594009 13563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:27.595613 13563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:27.596841 13563 hybrid_clock.cc:648] HybridClock initialized: now 1790614107596803 us; error 57 us; skew 500 ppm
I20260928 16:48:27.597208 13563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:27.605648 13563 webserver.cc:460] Webserver started at http://127.26.101.62:33247/ using document root <none> and password file <none>
I20260928 16:48:27.606508 13563 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:27.606622 13563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:27.606931 13563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:27.610255 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "110881520b554e4e9e5e46e72dd9c609"
format_stamp: "Formatted at 2026-09-28 16:48:27 on dist-test-slave-t0tw"
server_key: "20ef6726deb9ea7d8385931c3aeafb25"
server_key_iv: "cdf0a34392857140173177a928185dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:27.611161 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "110881520b554e4e9e5e46e72dd9c609"
format_stamp: "Formatted at 2026-09-28 16:48:27 on dist-test-slave-t0tw"
server_key: "20ef6726deb9ea7d8385931c3aeafb25"
server_key_iv: "cdf0a34392857140173177a928185dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:27.616549 13563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:27.620136 13577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:27.621541 13563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:48:27.621699 13563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "110881520b554e4e9e5e46e72dd9c609"
format_stamp: "Formatted at 2026-09-28 16:48:27 on dist-test-slave-t0tw"
server_key: "20ef6726deb9ea7d8385931c3aeafb25"
server_key_iv: "cdf0a34392857140173177a928185dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:27.621892 13563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:27.646695 13563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:27.647640 13563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:27.647894 13563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:27.665987 13563 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:40669
I20260928 16:48:27.666005 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:40669 every 8 connection(s)
I20260928 16:48:27.667374 13563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:27.671170 13630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:27.673379 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13563
I20260928 16:48:27.673568 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:27.674113 27028 external_mini_cluster.cc:1428] Setting key 0ac54d0cf493c057a9afb93610c0d10f
I20260928 16:48:27.683314 13630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: Bootstrap starting.
I20260928 16:48:27.686671 13630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:27.687736 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:27.676157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.26.101.62:40669)
0928 16:48:27.677186 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:27.677197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:27.677246 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:48:27.678929 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:27.678951 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:27.679480 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:27.679852 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:27.680448 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:27.681780 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:27.683199 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:27.683920 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:27.684090 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:27.685316 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:27.685324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:27.685338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:27.685888 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:27.685986 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:27.686009 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:27.686113 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:27.686349 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:48:27.686674 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":350,"threads_started":1}
I20260928 16:48:27.687811 13630 log.cc:824] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:27.690840 13630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: No bootstrap required, opened a new log
I20260928 16:48:27.698621 13630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } }
I20260928 16:48:27.699023 13630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:27.699147 13630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110881520b554e4e9e5e46e72dd9c609, State: Initialized, Role: FOLLOWER
I20260928 16:48:27.699717 13630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } }
I20260928 16:48:27.699954 13630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:27.700076 13630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:27.700227 13630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:27.702672 13630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } }
I20260928 16:48:27.703256 13630 leader_election.cc:302] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110881520b554e4e9e5e46e72dd9c609; no voters:
I20260928 16:48:27.703825 13630 leader_election.cc:288] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:27.703977 13635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:27.704666 13635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [term 1 LEADER]: Becoming Leader. State: Replica: 110881520b554e4e9e5e46e72dd9c609, State: Running, Role: LEADER
I20260928 16:48:27.705201 13635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } }
I20260928 16:48:27.705899 13630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:27.708072 13636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110881520b554e4e9e5e46e72dd9c609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } } }
I20260928 16:48:27.709156 13636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:27.708827 13637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110881520b554e4e9e5e46e72dd9c609. Latest consensus state: current_term: 1 leader_uuid: "110881520b554e4e9e5e46e72dd9c609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110881520b554e4e9e5e46e72dd9c609" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 40669 } } }
I20260928 16:48:27.709429 13637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:27.711586 13644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:27.716945 13644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:27.726603 13644 catalog_manager.cc:1347] Generated new cluster ID: e05ded24eb89454ba21975233f2d4a63
I20260928 16:48:27.726750 13644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:27.743577 13644 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:27.744927 13644 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:27.756500 13644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: Generated new TSK 0
I20260928 16:48:27.757375 13644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:27.765446 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--builtin_ntp_servers=127.26.101.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:28.109843 13654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:28.110147 13654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:28.110277 13654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:28.117868 13654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:28.117985 13654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:28.118147 13654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:28.127787 13654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:28.129393 13654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:28.131433 13654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:28.145044 13659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:28.145030 13660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:28.145264 13662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:28.145828 13654 server_base.cc:1034] running on GCE node
I20260928 16:48:28.146551 13654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:28.147639 13654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:28.148835 13654 hybrid_clock.cc:648] HybridClock initialized: now 1790614108148772 us; error 61 us; skew 500 ppm
I20260928 16:48:28.149149 13654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:28.151782 13654 webserver.cc:460] Webserver started at http://127.26.101.1:36389/ using document root <none> and password file <none>
I20260928 16:48:28.152504 13654 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:28.152607 13654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:28.152925 13654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:28.156153 13654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "060039e95bfa4ee5b2b7ea1fa731edcb"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "3aaddc638c4e317b83cf8ace2895feaf"
server_key_iv: "2f6d1ba595edf2ba75beb4e19f4efd15"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.157248 13654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "060039e95bfa4ee5b2b7ea1fa731edcb"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "3aaddc638c4e317b83cf8ace2895feaf"
server_key_iv: "2f6d1ba595edf2ba75beb4e19f4efd15"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.164489 13654 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 16:48:28.168218 13668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.169909 13654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:48:28.170063 13654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "060039e95bfa4ee5b2b7ea1fa731edcb"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "3aaddc638c4e317b83cf8ace2895feaf"
server_key_iv: "2f6d1ba595edf2ba75beb4e19f4efd15"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.170253 13654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:28.188364 13654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:28.189303 13654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:28.189574 13654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:28.190557 13654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:28.192122 13654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:28.192219 13654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.192315 13654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:28.192368 13654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.223778 13654 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:41303
I20260928 16:48:28.223837 13781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:41303 every 8 connection(s)
I20260928 16:48:28.225175 13654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:28.225611 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13654
I20260928 16:48:28.225791 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:28.226377 27028 external_mini_cluster.cc:1428] Setting key 1087f649a6641b51a9e5a0e402bfd485
I20260928 16:48:28.229539 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--builtin_ntp_servers=127.26.101.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:28.240653 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:28.228189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:33155 (local address 127.26.101.62:40669)
0928 16:48:28.230064 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:28.230071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:28.230219 (+ 148us) server_negotiation.cc:407] Connection header received
0928 16:48:28.232780 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:28.232786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:28.232865 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:28.232975 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:28.234513 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.235359 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.237573 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.237981 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.238084 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:28.239716 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:28.239723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:28.239727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:28.239753 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:28.239808 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:28.239833 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:28.239926 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:28.240186 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:48:28.240305 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1746,"thread_start_us":250,"threads_started":1}
I20260928 16:48:28.241657 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:28.228852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40669 (local address 127.26.101.1:33155)
0928 16:48:28.230031 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:28.230090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:28.232418 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:28.233271 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:28.233283 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:28.233408 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:28.234257 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:28.234285 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.235527 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.235532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:28.237400 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:28.237416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.238236 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.238243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:28.238535 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:28.239359 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:28.239361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:28.239406 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:28.239497 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:28.239965 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:28.239976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:28.239984 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:28.240151 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 16:48:28.240490 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":325,"threads_started":1}
I20260928 16:48:28.243180 13782 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40669
I20260928 16:48:28.243615 13782 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:28.244565 13782 heartbeater.cc:511] Master 127.26.101.62:40669 requested a full tablet report, sending...
I20260928 16:48:28.246842 13594 ts_manager.cc:194] Registered new tserver with Master: 060039e95bfa4ee5b2b7ea1fa731edcb (127.26.101.1:41303)
I20260928 16:48:28.248441 13594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.1:33155
W20260928 16:48:28.572412 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:28.572716 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:28.572842 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:28.581108 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:28.581245 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:28.581403 13786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:28.590929 13786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:28.592283 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:28.594135 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:28.607646 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:28.607671 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:28.607725 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:28.608623 13786 server_base.cc:1034] running on GCE node
I20260928 16:48:28.609233 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:28.610211 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:28.611433 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790614108611380 us; error 50 us; skew 500 ppm
I20260928 16:48:28.611760 13786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:28.614154 13786 webserver.cc:460] Webserver started at http://127.26.101.2:35497/ using document root <none> and password file <none>
I20260928 16:48:28.614858 13786 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:28.614955 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:28.615314 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:28.618248 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "414b69f5ab7c4c19ad8567e3a501ee12"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "b2621b96f4b99929d6740778358a809b"
server_key_iv: "2d6723a9c5e678aaf28971d58c13c909"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.619050 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "414b69f5ab7c4c19ad8567e3a501ee12"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "b2621b96f4b99929d6740778358a809b"
server_key_iv: "2d6723a9c5e678aaf28971d58c13c909"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.624410 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:28.628356 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.629772 13786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:48:28.629941 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "414b69f5ab7c4c19ad8567e3a501ee12"
format_stamp: "Formatted at 2026-09-28 16:48:28 on dist-test-slave-t0tw"
server_key: "b2621b96f4b99929d6740778358a809b"
server_key_iv: "2d6723a9c5e678aaf28971d58c13c909"
server_key_version: "encryptionkey@0"
I20260928 16:48:28.630152 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:28.651305 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:28.652300 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:28.652680 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:28.653723 13786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:28.655303 13786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:28.655412 13786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.655526 13786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:28.655597 13786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:28.692094 13786 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:41165
I20260928 16:48:28.692167 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:41165 every 8 connection(s)
I20260928 16:48:28.693410 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:28.700120 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13786
I20260928 16:48:28.700291 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:28.700740 27028 external_mini_cluster.cc:1428] Setting key 984831bcde93b303fc5e2d521fa0aab1
I20260928 16:48:28.704133 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--builtin_ntp_servers=127.26.101.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:28.706769 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:28.696011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:48921 (local address 127.26.101.62:40669)
0928 16:48:28.696186 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:28.696191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:28.697574 (+ 1383us) server_negotiation.cc:407] Connection header received
0928 16:48:28.699376 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:28.699381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:28.699458 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:28.699546 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:28.700834 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.701770 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.703977 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.704540 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.704664 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:28.705998 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:28.706004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:28.706008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:28.706032 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:28.706068 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:28.706082 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:28.706165 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:28.706379 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 16:48:28.706481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:48:28.707957 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:28.696446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40669 (local address 127.26.101.2:48921)
0928 16:48:28.697406 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:28.697450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:28.699099 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:28.699829 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:28.699839 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:28.699934 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:28.700626 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:28.700646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.701974 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.701980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:28.703823 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:28.703838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:28.704746 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:28.704751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:28.704986 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:28.705659 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:28.705661 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:28.705705 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:28.705806 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:28.706167 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:28.706175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:28.706181 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:48:28.706357 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 16:48:28.706751 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260928 16:48:28.709499 13914 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40669
I20260928 16:48:28.709848 13914 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:28.710759 13914 heartbeater.cc:511] Master 127.26.101.62:40669 requested a full tablet report, sending...
I20260928 16:48:28.712491 13594 ts_manager.cc:194] Registered new tserver with Master: 414b69f5ab7c4c19ad8567e3a501ee12 (127.26.101.2:41165)
I20260928 16:48:28.713670 13594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.2:48921
W20260928 16:48:29.039918 13917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:29.040223 13917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:29.040319 13917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:29.047614 13917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:29.047736 13917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:29.047849 13917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:29.057219 13917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:29.058499 13917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:29.060303 13917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:29.073750 13923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.073778 13922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:29.074003 13925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:29.074517 13917 server_base.cc:1034] running on GCE node
I20260928 16:48:29.075253 13917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:29.076229 13917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:29.077440 13917 hybrid_clock.cc:648] HybridClock initialized: now 1790614109077385 us; error 59 us; skew 500 ppm
I20260928 16:48:29.077754 13917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:29.080113 13917 webserver.cc:460] Webserver started at http://127.26.101.3:35299/ using document root <none> and password file <none>
I20260928 16:48:29.080852 13917 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:29.080965 13917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:29.081278 13917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:29.084306 13917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "8086a68d570944029783879905d06f43"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-t0tw"
server_key: "31b4833c0d7ccd95e0db86c1a32aa303"
server_key_iv: "5d85be439c8c4f00690330cabd057a4a"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.085122 13917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "8086a68d570944029783879905d06f43"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-t0tw"
server_key: "31b4833c0d7ccd95e0db86c1a32aa303"
server_key_iv: "5d85be439c8c4f00690330cabd057a4a"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.090509 13917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 16:48:29.094551 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.096032 13917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:48:29.096206 13917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "8086a68d570944029783879905d06f43"
format_stamp: "Formatted at 2026-09-28 16:48:29 on dist-test-slave-t0tw"
server_key: "31b4833c0d7ccd95e0db86c1a32aa303"
server_key_iv: "5d85be439c8c4f00690330cabd057a4a"
server_key_version: "encryptionkey@0"
I20260928 16:48:29.096437 13917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:29.120759 13917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:29.121721 13917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:29.122027 13917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:29.123054 13917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:29.124867 13917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:29.124965 13917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.125088 13917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:29.125164 13917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:29.156908 13917 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:38969
I20260928 16:48:29.156986 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:38969 every 8 connection(s)
I20260928 16:48:29.158264 13917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:29.161916 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 13917
I20260928 16:48:29.162117 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:29.162673 27028 external_mini_cluster.cc:1428] Setting key 1b9ea9162756e7bfcaf1aceb89008929
I20260928 16:48:29.172752 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.161376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:56549 (local address 127.26.101.62:40669)
0928 16:48:29.161573 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.161579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.163411 (+ 1832us) server_negotiation.cc:407] Connection header received
0928 16:48:29.165824 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.165830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.165916 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.166003 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.167162 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.167954 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.169581 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.170023 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.170137 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.171885 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.171890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.171894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:29.171916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.171953 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:29.171966 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.172045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.172290 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.172418 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:48:29.173954 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.162012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:40669 (local address 127.26.101.3:56549)
0928 16:48:29.163227 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.163283 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.165530 (+ 2247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.166257 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.166266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.166365 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:29.166959 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.166978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.168126 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.168131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.169422 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.169437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.170262 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.170270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.170520 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.171481 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:29.171485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:29.171532 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:29.171632 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:29.172065 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:29.172078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:29.172086 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.172262 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.172728 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":326,"threads_started":1}
I20260928 16:48:29.175539 14045 heartbeater.cc:347] Connected to a master server at 127.26.101.62:40669
I20260928 16:48:29.175846 14045 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:29.176848 14045 heartbeater.cc:511] Master 127.26.101.62:40669 requested a full tablet report, sending...
I20260928 16:48:29.178429 13594 ts_manager.cc:194] Registered new tserver with Master: 8086a68d570944029783879905d06f43 (127.26.101.3:38969)
I20260928 16:48:29.179330 13594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.3:56549
I20260928 16:48:29.191025 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:29.201490 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.195335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.26.101.62:40669)
0928 16:48:29.195556 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.195563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.195854 (+ 291us) server_negotiation.cc:407] Connection header received
0928 16:48:29.196026 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.196031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.196111 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.196197 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.196737 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.197482 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.198934 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.199424 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.199540 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.200776 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.200781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.200784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:29.200803 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.200837 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:29.200849 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.200975 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.201166 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.201226 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:29.205884 13594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:29.213056 13594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:29.237341 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.226008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:38969 (local address 127.0.0.1:50782)
0928 16:48:29.226967 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.227010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.227233 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.228611 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.228620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.228671 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:29.229174 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.229189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.231734 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.231752 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.234284 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.234300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.236318 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.236325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.236565 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.236625 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.236758 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.236988 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":1547,"threads_started":1}
I20260928 16:48:29.238905 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.226347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.26.101.3:38969)
0928 16:48:29.227900 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.227909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.227943 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:48:29.228046 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.228055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.228335 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.229429 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:29.229758 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.231471 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.234532 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.236076 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.237934 (+ 1858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.238258 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.238335 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.238522 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":1235,"threads_started":1}
I20260928 16:48:29.240675 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.226547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46010)
0928 16:48:29.231952 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.231972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.232119 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.233129 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.233134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.233161 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:29.233608 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.233621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.235786 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.235793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.238164 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.238179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.239838 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.239844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.240026 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.240061 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.240159 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.240313 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5230,"thread_start_us":5083,"threads_started":1}
I20260928 16:48:29.241555 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.227203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.26.101.1:41303)
0928 16:48:29.231583 (+ 4380us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.231593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.232270 (+ 677us) server_negotiation.cc:407] Connection header received
0928 16:48:29.232381 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.232389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.232658 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.232861 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:29.233871 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.235613 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.238410 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.239623 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.239750 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.240038 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.240953 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.241158 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4189,"thread_start_us":4072,"threads_started":1}
I20260928 16:48:29.243654 13979 tablet_service.cc:1467] Processing CreateTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 (DEFAULT_TABLE table=test-table [id=d889e9e89c944025b1dfec002ec03e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:29.245169 13979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe97dbae91d47a3af0a19b1fe7cb340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:29.248224 13716 tablet_service.cc:1467] Processing CreateTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 (DEFAULT_TABLE table=test-table [id=d889e9e89c944025b1dfec002ec03e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:29.249809 13716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe97dbae91d47a3af0a19b1fe7cb340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:29.250068 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.232070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:41165 (local address 127.0.0.1:57578)
0928 16:48:29.237643 (+ 5573us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.237663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.237776 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.244503 (+ 6727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.244508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.244540 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:29.244706 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.244717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.246346 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.246350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.247918 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.247933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.249275 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.249281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.249461 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.249494 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.249580 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.249730 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5451,"thread_start_us":11187,"threads_started":1}
I20260928 16:48:29.250149 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.232348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.26.101.2:41165)
0928 16:48:29.243863 (+ 11515us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.243871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.243902 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:48:29.244003 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.244010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.244275 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.244896 (+ 621us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:29.245144 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.246187 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.248116 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.249117 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.249237 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.249491 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.249649 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.249816 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11326,"thread_start_us":11212,"threads_started":1}
I20260928 16:48:29.252197 13782 heartbeater.cc:503] Master 127.26.101.62:40669 was elected leader, sending a full tablet report...
I20260928 16:48:29.254103 13848 tablet_service.cc:1467] Processing CreateTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 (DEFAULT_TABLE table=test-table [id=d889e9e89c944025b1dfec002ec03e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:29.255604 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe97dbae91d47a3af0a19b1fe7cb340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:29.261891 14063 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Bootstrap starting.
I20260928 16:48:29.264668 14064 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Bootstrap starting.
I20260928 16:48:29.265810 14063 tablet_bootstrap.cc:654] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:29.267038 14063 log.cc:824] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:29.268702 14064 tablet_bootstrap.cc:654] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:29.270079 14064 log.cc:824] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:29.270573 14066 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Bootstrap starting.
I20260928 16:48:29.271852 14063 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: No bootstrap required, opened a new log
I20260928 16:48:29.272167 14063 ts_tablet_manager.cc:1397] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 16:48:29.273779 14064 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: No bootstrap required, opened a new log
I20260928 16:48:29.274036 14064 ts_tablet_manager.cc:1397] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 16:48:29.274425 14066 tablet_bootstrap.cc:654] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:29.275704 14066 log.cc:824] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:29.278923 14066 tablet_bootstrap.cc:492] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: No bootstrap required, opened a new log
I20260928 16:48:29.279254 14066 ts_tablet_manager.cc:1397] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 16:48:29.282033 14064 raft_consensus.cc:348] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.282418 14064 raft_consensus.cc:374] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:29.282512 14064 raft_consensus.cc:729] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060039e95bfa4ee5b2b7ea1fa731edcb, State: Initialized, Role: FOLLOWER
I20260928 16:48:29.282224 14063 raft_consensus.cc:348] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.282658 14063 raft_consensus.cc:374] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:29.282774 14063 raft_consensus.cc:729] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8086a68d570944029783879905d06f43, State: Initialized, Role: FOLLOWER
I20260928 16:48:29.283180 14064 consensus_queue.cc:260] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.283382 14063 consensus_queue.cc:260] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.288726 14064 ts_tablet_manager.cc:1428] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 16:48:29.288995 14063 ts_tablet_manager.cc:1428] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260928 16:48:29.289379 14045 heartbeater.cc:503] Master 127.26.101.62:40669 was elected leader, sending a full tablet report...
I20260928 16:48:29.289861 14066 raft_consensus.cc:348] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.290333 14066 raft_consensus.cc:374] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:29.290441 14066 raft_consensus.cc:729] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414b69f5ab7c4c19ad8567e3a501ee12, State: Initialized, Role: FOLLOWER
I20260928 16:48:29.291042 14066 consensus_queue.cc:260] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.292152 13914 heartbeater.cc:503] Master 127.26.101.62:40669 was elected leader, sending a full tablet report...
I20260928 16:48:29.292826 14066 ts_tablet_manager.cc:1428] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 16:48:29.410310 14046 tablet.cc:2369] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:29.445479 13915 tablet.cc:2369] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:29.477872 13783 tablet.cc:2369] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:29.670179 14069 raft_consensus.cc:482] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:29.670519 14069 raft_consensus.cc:504] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.672622 14069 leader_election.cc:288] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8086a68d570944029783879905d06f43 (127.26.101.3:38969), 414b69f5ab7c4c19ad8567e3a501ee12 (127.26.101.2:41165)
I20260928 16:48:29.680936 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.673395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:58105 (local address 127.26.101.3:38969)
0928 16:48:29.673592 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.673599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.674259 (+ 660us) server_negotiation.cc:407] Connection header received
0928 16:48:29.674569 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.674574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.674662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.674746 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:29.675580 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.676438 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.678766 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.679980 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.680149 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.680222 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.680562 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.680687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:48:29.681006 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.672905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:38969 (local address 127.26.101.1:58105)
0928 16:48:29.674123 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.674145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.674396 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.674949 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.674953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.674979 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:29.675373 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.675384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.676566 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.676572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.678548 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.678564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.680185 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.680192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.680377 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.680427 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.680509 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.680644 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":4,"thread_start_us":1322,"threads_started":1}
I20260928 16:48:29.681200 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.673191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:55315 (local address 127.26.101.2:41165)
0928 16:48:29.673589 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.673595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.674698 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 16:48:29.674975 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.674979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.675055 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.675166 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:29.676029 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.677084 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.679290 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.680139 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.680288 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.680359 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.680794 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.680940 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260928 16:48:29.681721 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.672966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:41165 (local address 127.26.101.1:55315)
0928 16:48:29.674570 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.674594 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:29.674817 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:29.675341 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:29.675346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:29.675375 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:29.675829 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.675842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.677243 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.677248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.679106 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:29.679121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.680376 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.680381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:29.680564 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.680596 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:48:29.681374 (+ 778us) client_negotiation.cc:240] Negotiation successful
0928 16:48:29.681505 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":828,"threads_started":1}
I20260928 16:48:29.682379 13999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe97dbae91d47a3af0a19b1fe7cb340" candidate_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8086a68d570944029783879905d06f43" is_pre_election: true
I20260928 16:48:29.682686 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe97dbae91d47a3af0a19b1fe7cb340" candidate_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" is_pre_election: true
I20260928 16:48:29.682984 13999 raft_consensus.cc:2454] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060039e95bfa4ee5b2b7ea1fa731edcb in term 0.
I20260928 16:48:29.683269 13868 raft_consensus.cc:2454] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060039e95bfa4ee5b2b7ea1fa731edcb in term 0.
I20260928 16:48:29.683672 13672 leader_election.cc:302] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060039e95bfa4ee5b2b7ea1fa731edcb, 8086a68d570944029783879905d06f43; no voters:
I20260928 16:48:29.684172 14069 raft_consensus.cc:2781] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:29.684334 14069 raft_consensus.cc:482] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:29.684437 14069 raft_consensus.cc:3037] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:29.686568 14069 raft_consensus.cc:504] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.687201 14069 leader_election.cc:288] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [CANDIDATE]: Term 1 election: Requested vote from peers 8086a68d570944029783879905d06f43 (127.26.101.3:38969), 414b69f5ab7c4c19ad8567e3a501ee12 (127.26.101.2:41165)
I20260928 16:48:29.687677 13999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe97dbae91d47a3af0a19b1fe7cb340" candidate_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8086a68d570944029783879905d06f43"
I20260928 16:48:29.687731 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe97dbae91d47a3af0a19b1fe7cb340" candidate_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414b69f5ab7c4c19ad8567e3a501ee12"
I20260928 16:48:29.687891 13999 raft_consensus.cc:3037] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:29.687915 13868 raft_consensus.cc:3037] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:29.690083 13999 raft_consensus.cc:2454] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060039e95bfa4ee5b2b7ea1fa731edcb in term 1.
I20260928 16:48:29.690105 13868 raft_consensus.cc:2454] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060039e95bfa4ee5b2b7ea1fa731edcb in term 1.
I20260928 16:48:29.690583 13671 leader_election.cc:302] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060039e95bfa4ee5b2b7ea1fa731edcb, 414b69f5ab7c4c19ad8567e3a501ee12; no voters:
I20260928 16:48:29.690919 14069 raft_consensus.cc:2781] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:29.691423 14069 raft_consensus.cc:686] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 1 LEADER]: Becoming Leader. State: Replica: 060039e95bfa4ee5b2b7ea1fa731edcb, State: Running, Role: LEADER
I20260928 16:48:29.691922 14069 consensus_queue.cc:237] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } }
I20260928 16:48:29.696449 13594 catalog_manager.cc:5495] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 060039e95bfa4ee5b2b7ea1fa731edcb (127.26.101.1). New cstate: current_term: 1 leader_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060039e95bfa4ee5b2b7ea1fa731edcb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 41303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 } health_report { overall_health: UNKNOWN } } }
I20260928 16:48:29.713938 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.707842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.26.101.1:41303)
0928 16:48:29.708036 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.708044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.708464 (+ 420us) server_negotiation.cc:407] Connection header received
0928 16:48:29.708668 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.708672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.708765 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.708907 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:29.709384 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.710070 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.711589 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.712009 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.712124 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.712768 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:29.713342 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:29.713518 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.713663 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.713707 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:48:29.722508 13999 raft_consensus.cc:1261] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 1 FOLLOWER]: Refusing update from remote peer 060039e95bfa4ee5b2b7ea1fa731edcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:29.722512 13868 raft_consensus.cc:1261] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 1 FOLLOWER]: Refusing update from remote peer 060039e95bfa4ee5b2b7ea1fa731edcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:29.723529 14074 consensus_queue.cc:1035] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "414b69f5ab7c4c19ad8567e3a501ee12" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:29.723877 14069 consensus_queue.cc:1035] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8086a68d570944029783879905d06f43" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:29.748180 13594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:29.748431 13594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:29.751902 13594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110881520b554e4e9e5e46e72dd9c609: Sending DeleteTablet for 3 replicas of tablet dfe97dbae91d47a3af0a19b1fe7cb340
I20260928 16:48:29.753135 13979 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:29 UTC) from {username='slave'} at 127.0.0.1:50782
I20260928 16:48:29.753250 13716 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:29 UTC) from {username='slave'} at 127.0.0.1:46010
I20260928 16:48:29.753209 13848 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe97dbae91d47a3af0a19b1fe7cb340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:29 UTC) from {username='slave'} at 127.0.0.1:57578
I20260928 16:48:29.754990 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13654
I20260928 16:48:29.764544 14088 tablet_replica.cc:330] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: stopping tablet replica
I20260928 16:48:29.765043 14088 raft_consensus.cc:2229] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:29.765591 14088 raft_consensus.cc:2258] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:29.766419 14090 tablet_replica.cc:330] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: stopping tablet replica
I20260928 16:48:29.766868 14090 raft_consensus.cc:2229] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:29.767614 14090 raft_consensus.cc:2258] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:29.769356 14088 ts_tablet_manager.cc:1905] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:29.771301 14090 ts_tablet_manager.cc:1905] T dfe97dbae91d47a3af0a19b1fe7cb340 P 060039e95bfa4ee5b2b7ea1fa731edcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:29.772445 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.757698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.26.101.1:41303)
0928 16:48:29.757893 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.757900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.761868 (+ 3968us) server_negotiation.cc:407] Connection header received
0928 16:48:29.762851 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.762858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.762965 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.763044 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.764260 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.765517 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.767853 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.768521 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.768675 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.770344 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.770360 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.770377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:29.771169 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.771242 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:29.771283 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.771394 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.771959 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.772065 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:29.773303 13777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:29.773411 14089 tablet_replica.cc:330] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: stopping tablet replica
I20260928 16:48:29.773820 14089 raft_consensus.cc:2229] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:29.774524 14089 raft_consensus.cc:2258] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:29.776587 14089 ts_tablet_manager.cc:1905] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:29.777731 14088 ts_tablet_manager.cc:1918] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:29.777904 14088 log.cc:1196] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/dfe97dbae91d47a3af0a19b1fe7cb340
I20260928 16:48:29.778450 14088 ts_tablet_manager.cc:1939] T dfe97dbae91d47a3af0a19b1fe7cb340 P 8086a68d570944029783879905d06f43: Deleting consensus metadata
I20260928 16:48:29.780427 13581 catalog_manager.cc:4841] TS 8086a68d570944029783879905d06f43 (127.26.101.3:38969): tablet dfe97dbae91d47a3af0a19b1fe7cb340 (table test-table [id=d889e9e89c944025b1dfec002ec03e36]) successfully deleted
I20260928 16:48:29.782614 14089 ts_tablet_manager.cc:1918] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:29.782761 14089 log.cc:1196] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/dfe97dbae91d47a3af0a19b1fe7cb340
I20260928 16:48:29.783309 14089 ts_tablet_manager.cc:1939] T dfe97dbae91d47a3af0a19b1fe7cb340 P 414b69f5ab7c4c19ad8567e3a501ee12: Deleting consensus metadata
I20260928 16:48:29.785122 13580 catalog_manager.cc:4841] TS 414b69f5ab7c4c19ad8567e3a501ee12 (127.26.101.2:41165): tablet dfe97dbae91d47a3af0a19b1fe7cb340 (table test-table [id=d889e9e89c944025b1dfec002ec03e36]) successfully deleted
I20260928 16:48:29.879647 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13654
I20260928 16:48:29.897624 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13786
W20260928 16:48:29.897970 13580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.101.1:41303 (error 108)
I20260928 16:48:29.900244 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.899371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46036)
0928 16:48:29.899892 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.900052 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
W20260928 16:48:29.900499 13580 catalog_manager.cc:4584] TS 060039e95bfa4ee5b2b7ea1fa731edcb (127.26.101.1:41303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfe97dbae91d47a3af0a19b1fe7cb340: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
I20260928 16:48:29.904960 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.898492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.26.101.2:41165)
0928 16:48:29.898720 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:29.898726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:29.898748 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:48:29.898860 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:29.898865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:29.898980 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:29.899115 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:29.899585 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.900453 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.901823 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:29.902281 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:29.902400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:29.903507 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:29.903516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:29.903531 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:29.904152 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:29.904206 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:29.904228 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:29.904343 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:29.904487 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 16:48:29.904609 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:48:29.905694 13909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:29.954562 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.953961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46046)
0928 16:48:29.954248 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.954369 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 16:48:29.955790 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:29.955348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46052)
0928 16:48:29.955520 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:29.955611 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:48:29.995100 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13786
I20260928 16:48:30.011096 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13917
I20260928 16:48:30.016680 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.016232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46064)
0928 16:48:30.016451 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.016541 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 16:48:30.017735 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.017336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46076)
0928 16:48:30.017502 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.017585 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:48:30.017773 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.011981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.26.101.3:38969)
0928 16:48:30.012148 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:30.012152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:30.012172 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:30.012232 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:30.012236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:30.012340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:30.012431 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:30.012786 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.013552 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.014848 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:30.015305 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:30.015402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:30.016369 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:30.016380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:30.016400 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:30.017025 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:30.017079 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:30.017106 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:30.017215 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:30.017343 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 16:48:30.017456 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:48:30.018368 14039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:30.098577 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.097980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46078)
0928 16:48:30.098286 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.098402 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 16:48:30.100028 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:30.099474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:41303 (local address 127.0.0.1:46092)
0928 16:48:30.099732 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:30.099837 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.1:41303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 16:48:30.107445 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13917
I20260928 16:48:30.123194 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 13563
I20260928 16:48:30.124498 13625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:30.245591 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 13563
2026-09-28T16:48:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:30 dist-test-slave-t0tw krb5kdc[14100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:30 dist-test-slave-t0tw krb5kdc[14100](info): set up 2 sockets
Sep 28 16:48:30 dist-test-slave-t0tw krb5kdc[14100](info): commencing operation
krb5kdc: starting...
W20260928 16:48:31.287015 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.012s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:31 dist-test-slave-t0tw krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:48:31.543007 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37407
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:31.876986 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:31.877314 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:31.877434 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:31.884773 14113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:31.884891 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:31.884965 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:31.885035 14113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:31.885087 14113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:31.895202 14113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:31.896576 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:31.897908 14113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:31.909806 14121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:31.909780 14118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:31.909950 14119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:31.910641 14113 server_base.cc:1034] running on GCE node
I20260928 16:48:31.911605 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:31.913085 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:31.914328 14113 hybrid_clock.cc:648] HybridClock initialized: now 1790614111914268 us; error 44 us; skew 500 ppm
I20260928 16:48:31.914711 14113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:31 dist-test-slave-t0tw krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:31.917816 14113 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:48:31.919503 14113 webserver.cc:460] Webserver started at http://127.26.101.62:33531/ using document root <none> and password file <none>
I20260928 16:48:31.920187 14113 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:31.920301 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:31.920594 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:31.923635 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "6965f6f237b24beea76f0d723bfb21dd"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-t0tw"
server_key: "f129015f330c0005a076f9ebe9f5edcd"
server_key_iv: "b02fed29107e0f2834d4abb24225c18f"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.924381 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "6965f6f237b24beea76f0d723bfb21dd"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-t0tw"
server_key: "f129015f330c0005a076f9ebe9f5edcd"
server_key_iv: "b02fed29107e0f2834d4abb24225c18f"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.929555 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:31.933122 14128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:31.934451 14113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:31.934619 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "6965f6f237b24beea76f0d723bfb21dd"
format_stamp: "Formatted at 2026-09-28 16:48:31 on dist-test-slave-t0tw"
server_key: "f129015f330c0005a076f9ebe9f5edcd"
server_key_iv: "b02fed29107e0f2834d4abb24225c18f"
server_key_version: "encryptionkey@0"
I20260928 16:48:31.934806 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:31.961978 14113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:31.966187 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:31.966467 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:31.983884 14113 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37407
I20260928 16:48:31.983942 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37407 every 8 connection(s)
I20260928 16:48:31.985273 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:31.987497 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14113
I20260928 16:48:31.987707 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:31.988238 27028 external_mini_cluster.cc:1428] Setting key db032b7519262a2f8a5cd3c1c3dfc7e7
I20260928 16:48:31.989779 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:48:31 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:32.003758 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: Bootstrap starting.
I20260928 16:48:32.007009 14180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:32.008092 14180 log.cc:824] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:32.010977 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: No bootstrap required, opened a new log
I20260928 16:48:32.011461 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:31.990166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.26.101.62:37407)
0928 16:48:31.990952 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:31.990965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:31.991005 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:48:31.992656 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:31.992677 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:31.993051 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:31.993398 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:31.995201 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.996598 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.998412 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:31.999176 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:31.999334 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:32.004154 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:32.004163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:32.004179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:32.004220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:32.007300 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:32.008046 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:32.008053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:32.008060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:32.008243 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:32.008649 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:32.008654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:32.008657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:32.009223 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:32.009511 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:32.009966 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 16:48:32.010401 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
I20260928 16:48:32.019330 14180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } }
I20260928 16:48:32.019744 14180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:32.019842 14180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6965f6f237b24beea76f0d723bfb21dd, State: Initialized, Role: FOLLOWER
I20260928 16:48:32.020457 14180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } }
I20260928 16:48:32.020685 14180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:32.020821 14180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:32.020982 14180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:32.023288 14180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } }
I20260928 16:48:32.023780 14180 leader_election.cc:302] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6965f6f237b24beea76f0d723bfb21dd; no voters:
I20260928 16:48:32.024333 14180 leader_election.cc:288] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:32.024462 14185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:32.025076 14185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [term 1 LEADER]: Becoming Leader. State: Replica: 6965f6f237b24beea76f0d723bfb21dd, State: Running, Role: LEADER
I20260928 16:48:32.025621 14185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } }
I20260928 16:48:32.026327 14180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:32.028524 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6965f6f237b24beea76f0d723bfb21dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } } }
I20260928 16:48:32.028502 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6965f6f237b24beea76f0d723bfb21dd. Latest consensus state: current_term: 1 leader_uuid: "6965f6f237b24beea76f0d723bfb21dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6965f6f237b24beea76f0d723bfb21dd" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37407 } } }
I20260928 16:48:32.029076 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:32.029127 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:32.035669 14191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:32.040867 14191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:32.051144 14191 catalog_manager.cc:1347] Generated new cluster ID: 845ea2e647fc4712a6e9073a7b97a42e
I20260928 16:48:32.051263 14191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:32.066533 14191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:32.067914 14191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:32.078109 14191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: Generated new TSK 0
I20260928 16:48:32.078980 14191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.26.101.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
Entry for principal kudu/127.26.101.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab.
I20260928 16:48:32.150967 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:32.504686 14206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:32.504979 14206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:32.505096 14206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:32.513624 14206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:32.513754 14206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:32.513895 14206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:32.524176 14206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.1.keytab
--principal=kudu/127.26.101.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:32.525568 14206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:32.527041 14206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:32.540848 14215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:32.540932 14213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:32.541159 14212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:32.541967 14206 server_base.cc:1034] running on GCE node
I20260928 16:48:32.542615 14206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:32.543689 14206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:32.544914 14206 hybrid_clock.cc:648] HybridClock initialized: now 1790614112544858 us; error 47 us; skew 500 ppm
I20260928 16:48:32.545245 14206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:32 dist-test-slave-t0tw krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614112, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:32.548507 14206 init.cc:377] Logged in from keytab as kudu/127.26.101.1@KRBTEST.COM (short username kudu)
I20260928 16:48:32.550107 14206 webserver.cc:460] Webserver started at http://127.26.101.1:43825/ using document root <none> and password file <none>
I20260928 16:48:32.550803 14206 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:32.550910 14206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:32.551246 14206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:32.554252 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "1f4113675f8e42f7931cee2a101e2be3"
format_stamp: "Formatted at 2026-09-28 16:48:32 on dist-test-slave-t0tw"
server_key: "3a750faa5b1f9be59adf4c06110e5c82"
server_key_iv: "4e000a9124a2e4a279fe93e26cef9511"
server_key_version: "encryptionkey@0"
I20260928 16:48:32.554991 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4113675f8e42f7931cee2a101e2be3"
format_stamp: "Formatted at 2026-09-28 16:48:32 on dist-test-slave-t0tw"
server_key: "3a750faa5b1f9be59adf4c06110e5c82"
server_key_iv: "4e000a9124a2e4a279fe93e26cef9511"
server_key_version: "encryptionkey@0"
I20260928 16:48:32.560355 14206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:48:32.564014 14222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:32.565410 14206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:32.565567 14206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "1f4113675f8e42f7931cee2a101e2be3"
format_stamp: "Formatted at 2026-09-28 16:48:32 on dist-test-slave-t0tw"
server_key: "3a750faa5b1f9be59adf4c06110e5c82"
server_key_iv: "4e000a9124a2e4a279fe93e26cef9511"
server_key_version: "encryptionkey@0"
I20260928 16:48:32.565770 14206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:32.613726 14206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:32.617990 14206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:32.618280 14206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:32.619212 14206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:32.620605 14206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:32.620678 14206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:32.620779 14206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:32.620821 14206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:32.652205 14206 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:42729
I20260928 16:48:32.652261 14335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:42729 every 8 connection(s)
I20260928 16:48:32.653517 14206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:32.663475 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14206
I20260928 16:48:32.663674 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:32.664108 27028 external_mini_cluster.cc:1428] Setting key 105f25807135b1cfb0f5662c3b2476a8
Sep 28 16:48:32 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614112, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.1@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:32.673795 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:32.656117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:53965 (local address 127.26.101.62:37407)
0928 16:48:32.656560 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:32.656565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:32.657417 (+ 852us) server_negotiation.cc:407] Connection header received
0928 16:48:32.659344 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:32.659349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:32.659432 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:32.659527 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:32.661057 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:32.661781 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:32.663598 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:32.664070 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:32.664194 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:32.667922 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:32.667927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:32.667930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:32.667964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:32.670596 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:32.671432 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:32.671437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:32.671439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:32.671567 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:32.672006 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:32.672010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:32.672012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:32.672305 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:32.672441 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:32.673182 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 16:48:32.673422 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260928 16:48:32.674796 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:32.656581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37407 (local address 127.26.101.1:53965)
0928 16:48:32.657254 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:32.657297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:32.659013 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:32.659790 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:32.659798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:32.660320 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:32.660884 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:32.660902 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:32.661953 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:32.661957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:32.663409 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:32.663433 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:32.664327 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:32.664334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:32.664637 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:32.665340 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:32.665363 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:32.667696 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:32.670746 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:32.670756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:32.670775 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:32.671305 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:32.671683 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:32.671688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:32.671691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:32.671887 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:32.672431 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:32.672440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:32.672797 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 16:48:32.673043 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 16:48:32.673613 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
I20260928 16:48:32.676477 14336 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37407
I20260928 16:48:32.676822 14336 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:32.677709 14336 heartbeater.cc:511] Master 127.26.101.62:37407 requested a full tablet report, sending...
I20260928 16:48:32.680231 14145 ts_manager.cc:194] Registered new tserver with Master: 1f4113675f8e42f7931cee2a101e2be3 (127.26.101.1:42729)
I20260928 16:48:32.682225 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.1@KRBTEST.COM'} at 127.26.101.1:53965
WARNING: no policy specified for kudu/127.26.101.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
Entry for principal kudu/127.26.101.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab.
I20260928 16:48:32.722824 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.059409 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.059723 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.059823 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.067332 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.067481 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.067652 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:33.077809 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.2.keytab
--principal=kudu/127.26.101.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:33.079319 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.080617 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.093868 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.093909 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.093837 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.094713 14342 server_base.cc:1034] running on GCE node
I20260928 16:48:33.095417 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.096436 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.097638 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790614113097583 us; error 42 us; skew 500 ppm
I20260928 16:48:33.097998 14342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:33 dist-test-slave-t0tw krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.101435 14342 init.cc:377] Logged in from keytab as kudu/127.26.101.2@KRBTEST.COM (short username kudu)
I20260928 16:48:33.103152 14342 webserver.cc:460] Webserver started at http://127.26.101.2:32973/ using document root <none> and password file <none>
I20260928 16:48:33.103921 14342 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.104053 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.104367 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.107455 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "bb8e7f9f5da440f7a917986f636339e2"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "80f1a7073bec765af6cd90889b238347"
server_key_iv: "c0573c5302bfc6da6945385b1c7e029c"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.108323 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "bb8e7f9f5da440f7a917986f636339e2"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "80f1a7073bec765af6cd90889b238347"
server_key_iv: "c0573c5302bfc6da6945385b1c7e029c"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.113878 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:33.117769 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.119520 14342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 16:48:33.119702 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "bb8e7f9f5da440f7a917986f636339e2"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "80f1a7073bec765af6cd90889b238347"
server_key_iv: "c0573c5302bfc6da6945385b1c7e029c"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.119927 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.147352 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.152191 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.152479 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.153470 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:33.154948 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:33.155049 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.155210 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:33.155272 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.189236 14342 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:43011
I20260928 16:48:33.189333 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:43011 every 8 connection(s)
I20260928 16:48:33.190589 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:33.199503 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14342
I20260928 16:48:33.199693 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:33.200152 27028 external_mini_cluster.cc:1428] Setting key aadb8d2d11c65c70dce7baa2b109a96d
Sep 28 16:48:33 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.2@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:33.211782 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.193362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:35677 (local address 127.26.101.62:37407)
0928 16:48:33.193838 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.193843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.194707 (+ 864us) server_negotiation.cc:407] Connection header received
0928 16:48:33.196699 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.196705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.196790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.196889 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.198579 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.199390 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.201269 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.201774 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.201892 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.205655 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.205661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.205664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.205703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.208405 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.209249 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.209254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.209256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.209403 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.209877 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.209881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.209883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.210178 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.210347 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.211131 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.211398 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260928 16:48:33.212813 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.193791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37407 (local address 127.26.101.2:35677)
0928 16:48:33.194530 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.194577 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.196387 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.197169 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.197179 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.197758 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:33.198377 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.198397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.199549 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.199553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.201100 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.201113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.202023 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.202030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.202280 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.203030 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:33.203054 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:33.205370 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:33.208588 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.208599 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.208615 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.209099 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.209527 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.209533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.209536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.209755 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.210334 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:33.210343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:33.210713 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.210978 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.211596 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
I20260928 16:48:33.214351 14471 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37407
I20260928 16:48:33.214710 14471 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:33.215629 14471 heartbeater.cc:511] Master 127.26.101.62:37407 requested a full tablet report, sending...
I20260928 16:48:33.217370 14145 ts_manager.cc:194] Registered new tserver with Master: bb8e7f9f5da440f7a917986f636339e2 (127.26.101.2:43011)
I20260928 16:48:33.218251 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.2@KRBTEST.COM'} at 127.26.101.2:35677
WARNING: no policy specified for kudu/127.26.101.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
Entry for principal kudu/127.26.101.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab.
I20260928 16:48:33.259027 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:33.600179 14477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:33.600512 14477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:33.600633 14477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:33.608196 14477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:33.608347 14477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:33.608503 14477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:33.618701 14477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.3.keytab
--principal=kudu/127.26.101.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:33.620049 14477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:33.621376 14477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:33.634920 14482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.635207 14483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:33.635321 14485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:33.635855 14477 server_base.cc:1034] running on GCE node
I20260928 16:48:33.636547 14477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:33.637647 14477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:33.638881 14477 hybrid_clock.cc:648] HybridClock initialized: now 1790614113638804 us; error 65 us; skew 500 ppm
I20260928 16:48:33.639294 14477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:33 dist-test-slave-t0tw krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:33.642653 14477 init.cc:377] Logged in from keytab as kudu/127.26.101.3@KRBTEST.COM (short username kudu)
I20260928 16:48:33.644304 14477 webserver.cc:460] Webserver started at http://127.26.101.3:35703/ using document root <none> and password file <none>
I20260928 16:48:33.645031 14477 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:33.645169 14477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:33.645488 14477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:33.648521 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "dc103e07b70741c0b2d3f1597c1c85b5"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "843bf50a5925a5342d4d917100d47afc"
server_key_iv: "e438b0cb66efdeb5ca3cd25663944d75"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.649340 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "dc103e07b70741c0b2d3f1597c1c85b5"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "843bf50a5925a5342d4d917100d47afc"
server_key_iv: "e438b0cb66efdeb5ca3cd25663944d75"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.654915 14477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:48:33.658639 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.660208 14477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:33.660384 14477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "dc103e07b70741c0b2d3f1597c1c85b5"
format_stamp: "Formatted at 2026-09-28 16:48:33 on dist-test-slave-t0tw"
server_key: "843bf50a5925a5342d4d917100d47afc"
server_key_iv: "e438b0cb66efdeb5ca3cd25663944d75"
server_key_version: "encryptionkey@0"
I20260928 16:48:33.660579 14477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:33.679574 14477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:33.684062 14477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:33.684361 14477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:33.685407 14477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:33.686161 14336 heartbeater.cc:503] Master 127.26.101.62:37407 was elected leader, sending a full tablet report...
I20260928 16:48:33.687006 14477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:33.687119 14477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 16:48:33.687274 14477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:33.687340 14477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:33.719436 14477 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:43715
I20260928 16:48:33.719542 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:43715 every 8 connection(s)
I20260928 16:48:33.720778 14477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:33.725762 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14477
I20260928 16:48:33.725994 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:33.726552 27028 external_mini_cluster.cc:1428] Setting key ae11df20730f8f1e0767bb5b2afe50d6
Sep 28 16:48:33 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614113, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.3@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:48:33.742409 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.723778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:50949 (local address 127.26.101.62:37407)
0928 16:48:33.724279 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.724285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.725545 (+ 1260us) server_negotiation.cc:407] Connection header received
0928 16:48:33.727923 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.727929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.728012 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.728107 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.729699 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.730468 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.732087 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.732491 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.732604 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.736281 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.736289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.736292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.736330 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.739022 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.739916 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.739920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.739923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.740057 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.740529 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.740533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.740536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.740836 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.740991 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.741760 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.742006 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":211,"threads_started":1}
I20260928 16:48:33.743456 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.724422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37407 (local address 127.26.101.3:50949)
0928 16:48:33.725318 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.725385 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.727611 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.728372 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.728382 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.728939 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:33.729528 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.729546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.730626 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.730630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.731945 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.731958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.732730 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.732742 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.732969 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.733602 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:48:33.733630 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:33.736029 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:33.739200 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.739210 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.739232 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.739773 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.740170 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:48:33.740175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:48:33.740179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:48:33.740404 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:48:33.740974 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:33.740982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:33.741370 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.741631 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.742223 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260928 16:48:33.744781 14608 heartbeater.cc:347] Connected to a master server at 127.26.101.62:37407
I20260928 16:48:33.745085 14608 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:33.745849 14608 heartbeater.cc:511] Master 127.26.101.62:37407 requested a full tablet report, sending...
I20260928 16:48:33.747375 14145 ts_manager.cc:194] Registered new tserver with Master: dc103e07b70741c0b2d3f1597c1c85b5 (127.26.101.3:43715)
I20260928 16:48:33.748186 14145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.26.101.3@KRBTEST.COM'} at 127.26.101.3:50949
I20260928 16:48:33.754315 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:33.772197 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.758715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.26.101.62:37407)
0928 16:48:33.758880 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.758885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.759206 (+ 321us) server_negotiation.cc:407] Connection header received
0928 16:48:33.759350 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.759354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.759424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.759532 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:33.761138 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.761906 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.763422 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.763864 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.764011 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.766568 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:33.766574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:33.766578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:33.766617 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:33.769239 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.770104 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.770109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.770111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.770280 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:33.770588 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:33.770593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:33.770596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:33.770970 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:33.771220 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.771696 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.771913 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:48:33.776747 14145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:33.784016 14145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:33.809990 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.797195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:43011 (local address 127.0.0.1:55682)
0928 16:48:33.798829 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.798878 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.799106 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.801172 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.801180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.801210 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.801657 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.801674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.804205 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.804211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.806942 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.806978 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.808897 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.808903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.809126 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.809207 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.809335 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.809556 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":1127,"threads_started":1}
I20260928 16:48:33.810591 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.797515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.26.101.2:43011)
0928 16:48:33.800458 (+ 2943us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.800467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.800498 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:48:33.800616 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.800625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.800907 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.801891 (+ 984us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.802196 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.804006 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.807272 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.808670 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.809794 (+ 1124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.810017 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.810093 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.810278 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2771,"thread_start_us":3914,"threads_started":1}
I20260928 16:48:33.813879 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.797959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43715 (local address 127.0.0.1:33128)
0928 16:48:33.799838 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.799858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.799973 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.807134 (+ 7161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.807138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.807162 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.807602 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.807618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.809585 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.809590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.811737 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.811752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.813086 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.813092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.813287 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.813322 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.813395 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.813534 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754,"thread_start_us":1593,"threads_started":1}
I20260928 16:48:33.813956 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.798540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.26.101.3:43715)
0928 16:48:33.806400 (+ 7860us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.806409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.806450 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:48:33.806565 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.806573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.806880 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.807169 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.807841 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.809415 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.811899 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.812864 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.812984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.813224 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.813488 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.813658 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7684,"thread_start_us":7528,"threads_started":1}
I20260928 16:48:33.814814 14406 tablet_service.cc:1467] Processing CreateTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 (DEFAULT_TABLE table=test-table [id=f6f66bc04ab042ae9a30284240285f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.816515 14406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d1375dd7074bd8be7f26c3882a9960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.817291 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.799388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:42729 (local address 127.0.0.1:56258)
0928 16:48:33.806770 (+ 7382us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:33.806790 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:33.806927 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:33.809033 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:33.809038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:33.809064 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:33.809491 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.809503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.811794 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.811800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.814543 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:33.814559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.816473 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.816480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:33.816649 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.816685 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:48:33.816790 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:48:33.816934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7259,"thread_start_us":8022,"threads_started":1}
I20260928 16:48:33.817363 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:33.799995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.26.101.1:42729)
0928 16:48:33.808160 (+ 8165us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:33.808169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:33.808198 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:48:33.808303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:33.808311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:33.808558 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:33.808761 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:33.809683 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.811625 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.814756 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:33.816228 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:33.816389 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:33.816661 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:33.816848 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 16:48:33.817017 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7957,"thread_start_us":7770,"threads_started":1}
I20260928 16:48:33.818009 14542 tablet_service.cc:1467] Processing CreateTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 (DEFAULT_TABLE table=test-table [id=f6f66bc04ab042ae9a30284240285f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.819701 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d1375dd7074bd8be7f26c3882a9960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.821784 14270 tablet_service.cc:1467] Processing CreateTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 (DEFAULT_TABLE table=test-table [id=f6f66bc04ab042ae9a30284240285f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:33.823458 14270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d1375dd7074bd8be7f26c3882a9960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:33.830343 14627 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Bootstrap starting.
I20260928 16:48:33.833801 14627 tablet_bootstrap.cc:654] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.835028 14627 log.cc:824] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.838156 14627 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: No bootstrap required, opened a new log
I20260928 16:48:33.838447 14627 ts_tablet_manager.cc:1397] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 16:48:33.838994 14630 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Bootstrap starting.
I20260928 16:48:33.840481 14629 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Bootstrap starting.
I20260928 16:48:33.843371 14630 tablet_bootstrap.cc:654] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.844486 14629 tablet_bootstrap.cc:654] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:33.844795 14630 log.cc:824] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.845813 14629 log.cc:824] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:33.846489 14627 raft_consensus.cc:348] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.846890 14627 raft_consensus.cc:374] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.846988 14627 raft_consensus.cc:729] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8e7f9f5da440f7a917986f636339e2, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.847607 14627 consensus_queue.cc:260] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.848650 14630 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: No bootstrap required, opened a new log
I20260928 16:48:33.848996 14630 ts_tablet_manager.cc:1397] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 16:48:33.849318 14627 ts_tablet_manager.cc:1428] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 16:48:33.849738 14471 heartbeater.cc:503] Master 127.26.101.62:37407 was elected leader, sending a full tablet report...
I20260928 16:48:33.854246 14629 tablet_bootstrap.cc:492] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: No bootstrap required, opened a new log
I20260928 16:48:33.854540 14629 ts_tablet_manager.cc:1397] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 16:48:33.857475 14630 raft_consensus.cc:348] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.857874 14630 raft_consensus.cc:374] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.857978 14630 raft_consensus.cc:729] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4113675f8e42f7931cee2a101e2be3, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.858597 14630 consensus_queue.cc:260] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.860321 14630 ts_tablet_manager.cc:1428] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260928 16:48:33.862860 14629 raft_consensus.cc:348] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.863407 14629 raft_consensus.cc:374] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:33.863528 14629 raft_consensus.cc:729] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc103e07b70741c0b2d3f1597c1c85b5, State: Initialized, Role: FOLLOWER
I20260928 16:48:33.864233 14629 consensus_queue.cc:260] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:33.865492 14608 heartbeater.cc:503] Master 127.26.101.62:37407 was elected leader, sending a full tablet report...
I20260928 16:48:33.866195 14629 ts_tablet_manager.cc:1428] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 16:48:33.905998 14337 tablet.cc:2369] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:33.942723 14472 tablet.cc:2369] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:33.972532 14609 tablet.cc:2369] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:34.069440 14633 raft_consensus.cc:482] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:34.069756 14633 raft_consensus.cc:504] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:34.071560 14633 leader_election.cc:288] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4113675f8e42f7931cee2a101e2be3 (127.26.101.1:42729), dc103e07b70741c0b2d3f1597c1c85b5 (127.26.101.3:43715)
I20260928 16:48:34.080937 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.072219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:46253 (local address 127.26.101.3:43715)
0928 16:48:34.072411 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.072417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.072978 (+ 561us) server_negotiation.cc:407] Connection header received
0928 16:48:34.073110 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.073116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.073198 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.073313 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:34.074082 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.075241 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.078364 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.079663 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.079859 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.079938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.080514 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.080672 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:48:34.080941 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.072008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:46983 (local address 127.26.101.1:42729)
0928 16:48:34.072250 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.072256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.072905 (+ 649us) server_negotiation.cc:407] Connection header received
0928 16:48:34.073109 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.073116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.073198 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.073333 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:34.074081 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.075241 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.078364 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.079673 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.079859 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.079938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.080514 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.080663 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:48:34.081130 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.071890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:42729 (local address 127.26.101.2:46983)
0928 16:48:34.072771 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.072791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.072972 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.073345 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.073348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.073440 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:34.073918 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.073935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.075401 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.075406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.078103 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.078124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.080010 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.080018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.080260 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.080324 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.080439 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.080650 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":281,"spinlock_wait_cycles":25856,"thread_start_us":510,"threads_started":1}
I20260928 16:48:34.081535 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.071957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:43715 (local address 127.26.101.2:46253)
0928 16:48:34.072877 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:34.072892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:34.072984 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:34.073335 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:34.073340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:34.073367 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:34.073934 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.073944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.075399 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.075406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.078103 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:34.078124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.080010 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.080018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:34.080281 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.080324 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:48:34.080440 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:48:34.080650 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":193,"thread_start_us":612,"threads_started":1}
I20260928 16:48:34.082140 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d1375dd7074bd8be7f26c3882a9960" candidate_uuid: "bb8e7f9f5da440f7a917986f636339e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4113675f8e42f7931cee2a101e2be3" is_pre_election: true
I20260928 16:48:34.082459 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d1375dd7074bd8be7f26c3882a9960" candidate_uuid: "bb8e7f9f5da440f7a917986f636339e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" is_pre_election: true
I20260928 16:48:34.082649 14291 raft_consensus.cc:2454] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb8e7f9f5da440f7a917986f636339e2 in term 0.
I20260928 16:48:34.082974 14562 raft_consensus.cc:2454] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb8e7f9f5da440f7a917986f636339e2 in term 0.
I20260928 16:48:34.083432 14361 leader_election.cc:302] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4113675f8e42f7931cee2a101e2be3, bb8e7f9f5da440f7a917986f636339e2; no voters:
I20260928 16:48:34.083818 14633 raft_consensus.cc:2781] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:34.083947 14633 raft_consensus.cc:482] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:34.084044 14633 raft_consensus.cc:3037] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.086496 14633 raft_consensus.cc:504] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:34.087263 14633 leader_election.cc:288] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4113675f8e42f7931cee2a101e2be3 (127.26.101.1:42729), dc103e07b70741c0b2d3f1597c1c85b5 (127.26.101.3:43715)
I20260928 16:48:34.087836 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d1375dd7074bd8be7f26c3882a9960" candidate_uuid: "bb8e7f9f5da440f7a917986f636339e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc103e07b70741c0b2d3f1597c1c85b5"
I20260928 16:48:34.087847 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d1375dd7074bd8be7f26c3882a9960" candidate_uuid: "bb8e7f9f5da440f7a917986f636339e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4113675f8e42f7931cee2a101e2be3"
I20260928 16:48:34.088050 14562 raft_consensus.cc:3037] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.088061 14291 raft_consensus.cc:3037] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:34.090325 14562 raft_consensus.cc:2454] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8e7f9f5da440f7a917986f636339e2 in term 1.
I20260928 16:48:34.090525 14291 raft_consensus.cc:2454] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8e7f9f5da440f7a917986f636339e2 in term 1.
I20260928 16:48:34.090901 14359 leader_election.cc:302] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb8e7f9f5da440f7a917986f636339e2, dc103e07b70741c0b2d3f1597c1c85b5; no voters:
I20260928 16:48:34.091313 14633 raft_consensus.cc:2781] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:34.091768 14633 raft_consensus.cc:686] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 1 LEADER]: Becoming Leader. State: Replica: bb8e7f9f5da440f7a917986f636339e2, State: Running, Role: LEADER
I20260928 16:48:34.092252 14633 consensus_queue.cc:237] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } }
I20260928 16:48:34.096938 14145 catalog_manager.cc:5495] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb8e7f9f5da440f7a917986f636339e2 (127.26.101.2). New cstate: current_term: 1 leader_uuid: "bb8e7f9f5da440f7a917986f636339e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8e7f9f5da440f7a917986f636339e2" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 43011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 } health_report { overall_health: UNKNOWN } } }
I20260928 16:48:34.179307 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.174099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.26.101.2:43011)
0928 16:48:34.174276 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.174281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.174298 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:48:34.174390 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.174394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.174470 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.174609 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:34.174902 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.175724 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.177084 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.177489 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.177578 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.177961 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:34.178630 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:34.178788 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.178886 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.178976 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:48:34.188066 14291 raft_consensus.cc:1261] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 1 FOLLOWER]: Refusing update from remote peer bb8e7f9f5da440f7a917986f636339e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:34.188066 14562 raft_consensus.cc:1261] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 1 FOLLOWER]: Refusing update from remote peer bb8e7f9f5da440f7a917986f636339e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:34.189194 14639 consensus_queue.cc:1035] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4113675f8e42f7931cee2a101e2be3" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:34.189569 14633 consensus_queue.cc:1035] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc103e07b70741c0b2d3f1597c1c85b5" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:34.223345 14144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:34.223701 14144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:34.228005 14144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6965f6f237b24beea76f0d723bfb21dd: Sending DeleteTablet for 3 replicas of tablet d7d1375dd7074bd8be7f26c3882a9960
I20260928 16:48:34.229346 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:33128
I20260928 16:48:34.229450 14406 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:55682
I20260928 16:48:34.230494 14270 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d1375dd7074bd8be7f26c3882a9960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:34 UTC) from {username='kudu', principal='kudu/127.26.101.62@KRBTEST.COM'} at 127.0.0.1:56258
I20260928 16:48:34.231120 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14206
I20260928 16:48:34.243827 14653 tablet_replica.cc:330] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: stopping tablet replica
I20260928 16:48:34.247118 14652 tablet_replica.cc:330] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: stopping tablet replica
I20260928 16:48:34.247659 14653 raft_consensus.cc:2229] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:34.247818 14652 raft_consensus.cc:2229] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:34.248447 14653 raft_consensus.cc:2258] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:34.248621 14652 raft_consensus.cc:2258] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:34.251238 14652 ts_tablet_manager.cc:1905] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:34.253553 14653 ts_tablet_manager.cc:1905] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:34.259620 14651 tablet_replica.cc:330] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: stopping tablet replica
I20260928 16:48:34.260152 14651 raft_consensus.cc:2229] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:34.260712 14651 raft_consensus.cc:2258] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:34.261168 14652 ts_tablet_manager.cc:1918] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:34.261369 14652 log.cc:1196] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/d7d1375dd7074bd8be7f26c3882a9960
I20260928 16:48:34.261940 14652 ts_tablet_manager.cc:1939] T d7d1375dd7074bd8be7f26c3882a9960 P bb8e7f9f5da440f7a917986f636339e2: Deleting consensus metadata
I20260928 16:48:34.262979 14651 ts_tablet_manager.cc:1905] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:34.264128 14132 catalog_manager.cc:4841] TS bb8e7f9f5da440f7a917986f636339e2 (127.26.101.2:43011): tablet d7d1375dd7074bd8be7f26c3882a9960 (table test-table [id=f6f66bc04ab042ae9a30284240285f8e]) successfully deleted
I20260928 16:48:34.265854 14653 ts_tablet_manager.cc:1918] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:34.266059 14653 log.cc:1196] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/d7d1375dd7074bd8be7f26c3882a9960
I20260928 16:48:34.266659 14653 ts_tablet_manager.cc:1939] T d7d1375dd7074bd8be7f26c3882a9960 P 1f4113675f8e42f7931cee2a101e2be3: Deleting consensus metadata
Sep 28 16:48:34 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.1@KRBTEST.COM
I20260928 16:48:34.268846 14132 catalog_manager.cc:4841] TS 1f4113675f8e42f7931cee2a101e2be3 (127.26.101.1:42729): tablet d7d1375dd7074bd8be7f26c3882a9960 (table test-table [id=f6f66bc04ab042ae9a30284240285f8e]) successfully deleted
I20260928 16:48:34.271132 14651 ts_tablet_manager.cc:1918] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:34.271332 14651 log.cc:1196] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/d7d1375dd7074bd8be7f26c3882a9960
I20260928 16:48:34.271905 14651 ts_tablet_manager.cc:1939] T d7d1375dd7074bd8be7f26c3882a9960 P dc103e07b70741c0b2d3f1597c1c85b5: Deleting consensus metadata
I20260928 16:48:34.273931 14130 catalog_manager.cc:4841] TS dc103e07b70741c0b2d3f1597c1c85b5 (127.26.101.3:43715): tablet d7d1375dd7074bd8be7f26c3882a9960 (table test-table [id=f6f66bc04ab042ae9a30284240285f8e]) successfully deleted
I20260928 16:48:34.276965 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.232411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.26.101.1:42729)
0928 16:48:34.232602 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.232608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.242719 (+ 10111us) server_negotiation.cc:407] Connection header received
0928 16:48:34.244410 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.244417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.244544 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.244632 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.248716 (+ 4084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.249991 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.255221 (+ 5230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.255922 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.256038 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.269016 (+ 12978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.269032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.269050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:34.269100 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.273375 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.274588 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.274597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.274599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.274772 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.275293 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.275297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.275299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.275669 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.275930 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.276394 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.276597 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:48:34.277663 14331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:34.371803 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14206
I20260928 16:48:34.388279 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14342
Sep 28 16:48:34 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.2@KRBTEST.COM
I20260928 16:48:34.407398 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.389223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.26.101.2:43011)
0928 16:48:34.389392 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.389396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.389415 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:48:34.389487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.389490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.389552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.389634 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.391359 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.392166 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.394119 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.394606 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.394704 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.400033 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.400056 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.400078 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:34.400141 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.404305 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.405120 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.405130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.405134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.405323 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.405658 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.405663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.405667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.406128 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.406467 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.406736 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.406974 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:48:34.408274 14466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:34.523057 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14342
I20260928 16:48:34.540257 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14477
Sep 28 16:48:34 dist-test-slave-t0tw krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.3@KRBTEST.COM
I20260928 16:48:34.556181 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:34.541037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.26.101.3:43715)
0928 16:48:34.541243 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:34.541248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:34.541268 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:34.541479 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:34.541484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:34.541558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:34.541651 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:34.543001 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.543851 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.545279 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:34.545710 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:34.545820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:34.550413 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:34.550431 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:34.550450 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:34.550500 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:34.553339 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.554116 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.554122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.554125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.554308 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:34.554661 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:34.554665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:34.554668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:34.555119 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:34.555465 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:34.555663 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 16:48:34.555841 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:48:34.556833 14603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:34.646049 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14477
I20260928 16:48:34.661095 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14113
I20260928 16:48:34.662196 14175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:34.774514 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14113
2026-09-28T16:48:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4529 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T16:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:34Z Disabled control of system clock
I20260928 16:48:34.851066 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:38761
--webserver_interface=127.26.101.62
--webserver_port=0
--builtin_ntp_servers=127.26.101.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:35.187211 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:35.187525 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:35.187608 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:35.194991 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:35.195159 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:35.195222 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:35.195283 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:35.195330 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:35.205001 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:38761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:35.206282 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:35.208113 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:35.219766 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:35.219801 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:35.219847 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:35.220691 14662 server_base.cc:1034] running on GCE node
I20260928 16:48:35.221467 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:35.222916 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:35.224134 14662 hybrid_clock.cc:648] HybridClock initialized: now 1790614115224060 us; error 63 us; skew 500 ppm
I20260928 16:48:35.224478 14662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:35.226907 14662 webserver.cc:460] Webserver started at http://127.26.101.62:32973/ using document root <none> and password file <none>
I20260928 16:48:35.227669 14662 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:35.227768 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:35.228053 14662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:35.231038 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "ef93c0e870ca4bc38b6c6195a49b514e"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "78e6071b56db705f0a947d29a986a8ff"
server_key_iv: "a0342661627d4fe0fd50fd3dfc96ffd0"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.231868 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "ef93c0e870ca4bc38b6c6195a49b514e"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "78e6071b56db705f0a947d29a986a8ff"
server_key_iv: "a0342661627d4fe0fd50fd3dfc96ffd0"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.237267 14662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:35.240897 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:35.242266 14662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:35.242447 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "ef93c0e870ca4bc38b6c6195a49b514e"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "78e6071b56db705f0a947d29a986a8ff"
server_key_iv: "a0342661627d4fe0fd50fd3dfc96ffd0"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.242655 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:35.259620 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:35.260574 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:35.260864 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:35.278970 14662 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:38761
I20260928 16:48:35.278990 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:38761 every 8 connection(s)
I20260928 16:48:35.280445 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:35.284291 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:35.286358 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14662
I20260928 16:48:35.286551 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:35.287122 27028 external_mini_cluster.cc:1428] Setting key 52cc2d317cf15a7520be570383ac82d5
I20260928 16:48:35.296434 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: Bootstrap starting.
I20260928 16:48:35.299844 14729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:35.300851 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.289103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.26.101.62:38761)
0928 16:48:35.290273 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.290286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.290337 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:48:35.292130 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.292155 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.292633 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.292988 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:35.293595 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.294938 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.296313 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.297057 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.297236 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.298469 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:35.298476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:35.298492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:35.299033 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:35.299172 (+ 139us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:35.299197 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:35.299323 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.299536 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.299873 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":374,"threads_started":1}
I20260928 16:48:35.300947 14729 log.cc:824] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:35.304095 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: No bootstrap required, opened a new log
I20260928 16:48:35.311617 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } }
I20260928 16:48:35.311995 14729 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:35.312107 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef93c0e870ca4bc38b6c6195a49b514e, State: Initialized, Role: FOLLOWER
I20260928 16:48:35.312659 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } }
I20260928 16:48:35.312889 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:35.313014 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:35.313156 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:35.315680 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } }
I20260928 16:48:35.316190 14729 leader_election.cc:302] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef93c0e870ca4bc38b6c6195a49b514e; no voters:
I20260928 16:48:35.316761 14729 leader_election.cc:288] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:35.316901 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:35.317749 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [term 1 LEADER]: Becoming Leader. State: Replica: ef93c0e870ca4bc38b6c6195a49b514e, State: Running, Role: LEADER
I20260928 16:48:35.318329 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } }
I20260928 16:48:35.318966 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:35.321506 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef93c0e870ca4bc38b6c6195a49b514e. Latest consensus state: current_term: 1 leader_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } } }
I20260928 16:48:35.321930 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:35.323036 14741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:35.322990 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef93c0e870ca4bc38b6c6195a49b514e" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 38761 } } }
I20260928 16:48:35.323514 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:35.328560 14741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:35.338394 14741 catalog_manager.cc:1347] Generated new cluster ID: 4b7b056fedc64418b690c0d57b607cec
I20260928 16:48:35.338533 14741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:35.350664 14741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:35.352531 14741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:35.366974 14741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: Generated new TSK 0
I20260928 16:48:35.368144 14741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:35.372483 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.1:0
--local_ip_for_outbound_sockets=127.26.101.1
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--builtin_ntp_servers=127.26.101.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:48:35.707139 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:35.707453 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:35.707551 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:35.714854 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:35.714968 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:35.715098 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.1
I20260928 16:48:35.724756 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.101.1
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.1
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:35.726151 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:35.728012 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:35.740587 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:35.740540 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:35.740504 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:35.741863 14753 server_base.cc:1034] running on GCE node
I20260928 16:48:35.742487 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:35.743602 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:35.744836 14753 hybrid_clock.cc:648] HybridClock initialized: now 1790614115744775 us; error 58 us; skew 500 ppm
I20260928 16:48:35.745160 14753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:35.747686 14753 webserver.cc:460] Webserver started at http://127.26.101.1:39089/ using document root <none> and password file <none>
I20260928 16:48:35.748427 14753 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:35.748534 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:35.748869 14753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:35.752068 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data/instance:
uuid: "e2c630cd455f496fa7ad4eb06b5292fb"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "db58913ad8db9c2cdb0c009fbefa440d"
server_key_iv: "19ca9d0a0c791579e0fbbfaa99ea8f58"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.752933 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c630cd455f496fa7ad4eb06b5292fb"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "db58913ad8db9c2cdb0c009fbefa440d"
server_key_iv: "19ca9d0a0c791579e0fbbfaa99ea8f58"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.758661 14753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:35.762748 14767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:35.764338 14753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:48:35.764577 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal
uuid: "e2c630cd455f496fa7ad4eb06b5292fb"
format_stamp: "Formatted at 2026-09-28 16:48:35 on dist-test-slave-t0tw"
server_key: "db58913ad8db9c2cdb0c009fbefa440d"
server_key_iv: "19ca9d0a0c791579e0fbbfaa99ea8f58"
server_key_version: "encryptionkey@0"
I20260928 16:48:35.764830 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:35.779906 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:35.780777 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:35.781075 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:35.782115 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:35.783715 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:35.783808 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:35.783917 14753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:35.783993 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:35.814991 14753 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.1:44191
I20260928 16:48:35.815026 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.1:44191 every 8 connection(s)
I20260928 16:48:35.816365 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/data/info.pb
I20260928 16:48:35.817608 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14753
I20260928 16:48:35.817795 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal/instance
I20260928 16:48:35.818500 27028 external_mini_cluster.cc:1428] Setting key f172bb10f2f1b606f1262ab594d06e27
I20260928 16:48:35.821612 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.2:0
--local_ip_for_outbound_sockets=127.26.101.2
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--builtin_ntp_servers=127.26.101.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:35.830215 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.819240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:43039 (local address 127.26.101.62:38761)
0928 16:48:35.819767 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:35.819773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:35.821091 (+ 1318us) server_negotiation.cc:407] Connection header received
0928 16:48:35.823135 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:35.823141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:35.823232 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:35.823342 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:35.824640 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.825567 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.827386 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.827959 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.828083 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.829460 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:35.829466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:35.829471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:35.829495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:35.829542 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:35.829555 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:35.829623 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:35.829884 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 16:48:35.829972 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260928 16:48:35.831674 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:35.819946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:38761 (local address 127.26.101.1:43039)
0928 16:48:35.820939 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:35.820977 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:35.822750 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:35.823605 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:35.823628 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:35.823727 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:35.824451 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.824481 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.825738 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.825742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.827218 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:35.827233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:35.828167 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:35.828172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:35.828418 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:35.829110 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:35.829113 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:35.829157 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:35.829241 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:35.829647 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:35.829658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:35.829666 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:35.829857 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 16:48:35.830290 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":309,"threads_started":1}
I20260928 16:48:35.833420 14881 heartbeater.cc:347] Connected to a master server at 127.26.101.62:38761
I20260928 16:48:35.833767 14881 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:35.834679 14881 heartbeater.cc:511] Master 127.26.101.62:38761 requested a full tablet report, sending...
I20260928 16:48:35.837093 14693 ts_manager.cc:194] Registered new tserver with Master: e2c630cd455f496fa7ad4eb06b5292fb (127.26.101.1:44191)
I20260928 16:48:35.839116 14693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.1:43039
W20260928 16:48:36.157091 14885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:36.157495 14885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:36.157635 14885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:36.167567 14885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:36.167701 14885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:36.167862 14885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.2
I20260928 16:48:36.177413 14885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.101.2
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.2
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:36.178781 14885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:36.180624 14885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:36.195046 14890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.195144 14891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.195274 14893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:36.195861 14885 server_base.cc:1034] running on GCE node
I20260928 16:48:36.196484 14885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:36.197453 14885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:36.198650 14885 hybrid_clock.cc:648] HybridClock initialized: now 1790614116198603 us; error 39 us; skew 500 ppm
I20260928 16:48:36.198961 14885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:36.201412 14885 webserver.cc:460] Webserver started at http://127.26.101.2:34207/ using document root <none> and password file <none>
I20260928 16:48:36.202145 14885 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:36.202255 14885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:36.202564 14885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:36.205513 14885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data/instance:
uuid: "122c11fc6c894181a53cde61b730e9cb"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "c2aac4006dad29210629d98085ed9fbb"
server_key_iv: "ebf4a4f95edf8f79a6edd9674a1313d0"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.206279 14885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance:
uuid: "122c11fc6c894181a53cde61b730e9cb"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "c2aac4006dad29210629d98085ed9fbb"
server_key_iv: "ebf4a4f95edf8f79a6edd9674a1313d0"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.211552 14885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:48:36.215063 14899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.216535 14885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:36.216696 14885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal
uuid: "122c11fc6c894181a53cde61b730e9cb"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "c2aac4006dad29210629d98085ed9fbb"
server_key_iv: "ebf4a4f95edf8f79a6edd9674a1313d0"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.216893 14885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:36.238304 14885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:36.239229 14885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:36.239530 14885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:36.240552 14885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:36.242059 14885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:36.242149 14885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.242268 14885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:36.242338 14885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.272852 14885 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.2:38845
I20260928 16:48:36.272934 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.2:38845 every 8 connection(s)
I20260928 16:48:36.274115 14885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/data/info.pb
I20260928 16:48:36.281731 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 14885
I20260928 16:48:36.281900 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal/instance
I20260928 16:48:36.282366 27028 external_mini_cluster.cc:1428] Setting key e880ee2a4787030b2c03f3aaafc7b591
I20260928 16:48:36.285146 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.101.3:0
--local_ip_for_outbound_sockets=127.26.101.3
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--builtin_ntp_servers=127.26.101.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:48:36.288777 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.276637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.2:44503 (local address 127.26.101.62:38761)
0928 16:48:36.276869 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.276876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.278286 (+ 1410us) server_negotiation.cc:407] Connection header received
0928 16:48:36.280765 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.280770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.280838 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.280927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:36.282283 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.283339 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.285708 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.286305 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.286431 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.288001 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:36.288007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:36.288011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:36.288032 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:36.288072 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:36.288085 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:36.288170 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.288376 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.288475 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:48:36.290019 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.277150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:38761 (local address 127.26.101.2:44503)
0928 16:48:36.278067 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.278116 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.280450 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.281153 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.281164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.281292 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:36.282061 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.282086 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.283541 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.283547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.285538 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.285553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.286535 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.286542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.286833 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.287629 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:36.287633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:36.287696 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:36.287802 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:36.288173 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:36.288182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:36.288196 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.288358 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.288756 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260928 16:48:36.291556 15013 heartbeater.cc:347] Connected to a master server at 127.26.101.62:38761
I20260928 16:48:36.291886 15013 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:36.292743 15013 heartbeater.cc:511] Master 127.26.101.62:38761 requested a full tablet report, sending...
I20260928 16:48:36.294268 14693 ts_manager.cc:194] Registered new tserver with Master: 122c11fc6c894181a53cde61b730e9cb (127.26.101.2:38845)
I20260928 16:48:36.295364 14693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.2:44503
W20260928 16:48:36.610769 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:36.611068 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:36.611208 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:36.618553 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:36.618664 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:36.618772 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.101.3
I20260928 16:48:36.628895 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.101.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.101.3
--webserver_port=0
--tserver_master_addrs=127.26.101.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.101.3
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:36.630151 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:36.632016 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:36.645215 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.645242 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:36.645457 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:36.645896 15016 server_base.cc:1034] running on GCE node
I20260928 16:48:36.646560 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:36.647637 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:36.648844 15016 hybrid_clock.cc:648] HybridClock initialized: now 1790614116648788 us; error 46 us; skew 500 ppm
I20260928 16:48:36.649142 15016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 16:48:36.651438 15016 webserver.cc:460] Webserver started at http://127.26.101.3:35077/ using document root <none> and password file <none>
I20260928 16:48:36.652113 15016 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:36.652211 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:36.652482 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:36.655454 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data/instance:
uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "3a1f2eec2da75508bace2e642a6d1457"
server_key_iv: "7696d9151ac329e0ba45e7bc59b370e5"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.656237 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance:
uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "3a1f2eec2da75508bace2e642a6d1457"
server_key_iv: "7696d9151ac329e0ba45e7bc59b370e5"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.661540 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:36.665050 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.666450 15016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:48:36.666599 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal
uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b"
format_stamp: "Formatted at 2026-09-28 16:48:36 on dist-test-slave-t0tw"
server_key: "3a1f2eec2da75508bace2e642a6d1457"
server_key_iv: "7696d9151ac329e0ba45e7bc59b370e5"
server_key_version: "encryptionkey@0"
I20260928 16:48:36.666811 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:36.707512 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:36.708391 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:36.708696 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:36.709719 15016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:48:36.711253 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:48:36.711346 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.711457 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:48:36.711529 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:36.745144 15016 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.3:37675
I20260928 16:48:36.745183 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.3:37675 every 8 connection(s)
I20260928 16:48:36.746510 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/data/info.pb
I20260928 16:48:36.752997 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15016
I20260928 16:48:36.753168 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal/instance
I20260928 16:48:36.753638 27028 external_mini_cluster.cc:1428] Setting key 103504c6078d7f2290e4044e00473e7d
I20260928 16:48:36.759531 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.749044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.3:39573 (local address 127.26.101.62:38761)
0928 16:48:36.749313 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.749317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.750633 (+ 1316us) server_negotiation.cc:407] Connection header received
0928 16:48:36.752552 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.752559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.752681 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.752809 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:36.753983 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.754889 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.756813 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.757243 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.757356 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.758688 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:36.758693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:36.758696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:36.758715 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:36.758747 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:36.758758 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:36.758844 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.759113 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.759213 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 16:48:36.760514 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.749577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:38761 (local address 127.26.101.3:39573)
0928 16:48:36.750477 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.750514 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.752235 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.753002 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.753010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.753165 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:48:36.753800 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.753822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.755041 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.755045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.756668 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.756681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.757472 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.757481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.757694 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.758364 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:48:36.758366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:48:36.758409 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:48:36.758500 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:48:36.758856 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:48:36.758867 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:48:36.758875 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.759030 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.759476 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260928 16:48:36.761793 15144 heartbeater.cc:347] Connected to a master server at 127.26.101.62:38761
I20260928 16:48:36.762082 15144 heartbeater.cc:464] Registering TS with master...
I20260928 16:48:36.762843 15144 heartbeater.cc:511] Master 127.26.101.62:38761 requested a full tablet report, sending...
I20260928 16:48:36.764384 14693 ts_manager.cc:194] Registered new tserver with Master: 8ec3e36f36274f6ca48ee6fd3b3a098b (127.26.101.3:37675)
I20260928 16:48:36.765179 14693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.101.3:39573
I20260928 16:48:36.769021 27028 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:48:36.778932 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.773097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.26.101.62:38761)
0928 16:48:36.773306 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.773313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.773708 (+ 395us) server_negotiation.cc:407] Connection header received
0928 16:48:36.773833 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.773837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.773901 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.773981 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:36.774420 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.775160 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.776656 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.777066 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.777159 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.778232 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:36.778238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:36.778241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:36.778260 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:36.778300 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:36.778312 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:36.778382 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.778549 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.778629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:48:36.782578 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:48:36.789494 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:48:36.812762 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.801502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:38845 (local address 127.0.0.1:35058)
0928 16:48:36.803451 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.803494 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.803674 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.804793 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.804804 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.804839 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:36.805291 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.805305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.807407 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.807413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.809805 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.809826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.811732 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.811738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.811962 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.812029 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.812169 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.812355 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"mutex_wait_us":4,"thread_start_us":1487,"threads_started":1}
I20260928 16:48:36.813993 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.801733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.26.101.2:38845)
0928 16:48:36.804098 (+ 2365us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.804107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.804137 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:48:36.804249 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.804257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.804538 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.805517 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:36.805751 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.807259 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.810009 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.811520 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.811659 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.811911 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.813351 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.813534 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":2080,"threads_started":1}
I20260928 16:48:36.814148 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.803857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:37675 (local address 127.0.0.1:38516)
0928 16:48:36.806143 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.806159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.806262 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.807354 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.807359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.807382 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:36.807861 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.807877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.809621 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.809627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.811298 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.811312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.813399 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.813405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.813568 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.813599 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.813667 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.813789 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2175,"mutex_wait_us":45,"thread_start_us":902,"threads_started":1}
I20260928 16:48:36.815181 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.803521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.26.101.3:37675)
0928 16:48:36.806778 (+ 3257us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.806787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.806818 (+ 31us) server_negotiation.cc:407] Connection header received
0928 16:48:36.806908 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.806915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.807163 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.807358 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:36.808036 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.809497 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.811487 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.812792 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.814478 (+ 1686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.814691 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.814748 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.814861 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3085,"thread_start_us":2935,"threads_started":1}
I20260928 16:48:36.815613 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.801679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.1:44191 (local address 127.0.0.1:42194)
0928 16:48:36.804231 (+ 2552us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:36.804249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:36.804349 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:36.807702 (+ 3353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:36.807707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:36.807736 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:36.808192 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.808209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.809953 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.809958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.812043 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:36.812056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.813874 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.813880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:36.814090 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.815112 (+ 1022us) client_negotiation.cc:769] Sending connection context
0928 16:48:36.815198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:48:36.815340 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2422,"thread_start_us":2745,"threads_started":1}
I20260928 16:48:36.816457 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:36.802195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.26.101.1:44191)
0928 16:48:36.807115 (+ 4920us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:36.807124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:36.807150 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:48:36.807236 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:36.807240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:36.807437 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:36.807688 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:36.808397 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.809814 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.812239 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:36.813403 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:36.813525 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:36.813739 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:36.815908 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0928 16:48:36.816075 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4758,"thread_start_us":4625,"threads_started":1}
I20260928 16:48:36.818310 14947 tablet_service.cc:1467] Processing CreateTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (DEFAULT_TABLE table=test-table [id=a661edc93da340628be1a35055259f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:36.818986 15078 tablet_service.cc:1467] Processing CreateTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (DEFAULT_TABLE table=test-table [id=a661edc93da340628be1a35055259f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:36.819952 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:36.820559 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:36.820225 14815 tablet_service.cc:1467] Processing CreateTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (DEFAULT_TABLE table=test-table [id=a661edc93da340628be1a35055259f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:48:36.821468 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:36.835587 15162 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Bootstrap starting.
I20260928 16:48:36.837834 15163 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Bootstrap starting.
I20260928 16:48:36.838732 15164 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Bootstrap starting.
I20260928 16:48:36.841187 15162 tablet_bootstrap.cc:654] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:36.841856 15163 tablet_bootstrap.cc:654] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:36.842499 15162 log.cc:824] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:36.842772 15164 tablet_bootstrap.cc:654] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:36.842868 14881 heartbeater.cc:503] Master 127.26.101.62:38761 was elected leader, sending a full tablet report...
I20260928 16:48:36.842846 15163 log.cc:824] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:36.844134 15164 log.cc:824] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:36.849328 15163 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: No bootstrap required, opened a new log
I20260928 16:48:36.849712 15163 ts_tablet_manager.cc:1397] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260928 16:48:36.853323 15164 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: No bootstrap required, opened a new log
I20260928 16:48:36.853603 15164 ts_tablet_manager.cc:1397] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260928 16:48:36.853919 15162 tablet_bootstrap.cc:492] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: No bootstrap required, opened a new log
I20260928 16:48:36.854207 15162 ts_tablet_manager.cc:1397] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.009s
I20260928 16:48:36.859648 15163 raft_consensus.cc:348] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.860316 15163 raft_consensus.cc:374] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:36.860477 15163 raft_consensus.cc:729] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122c11fc6c894181a53cde61b730e9cb, State: Initialized, Role: FOLLOWER
I20260928 16:48:36.861163 15164 raft_consensus.cc:348] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.861250 15163 consensus_queue.cc:260] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.861536 15164 raft_consensus.cc:374] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:36.861649 15164 raft_consensus.cc:729] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c630cd455f496fa7ad4eb06b5292fb, State: Initialized, Role: FOLLOWER
I20260928 16:48:36.862365 15164 consensus_queue.cc:260] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.862506 15013 heartbeater.cc:503] Master 127.26.101.62:38761 was elected leader, sending a full tablet report...
I20260928 16:48:36.864168 15164 ts_tablet_manager.cc:1428] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260928 16:48:36.864492 15162 raft_consensus.cc:348] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.865011 15162 raft_consensus.cc:374] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:36.865119 15162 raft_consensus.cc:729] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec3e36f36274f6ca48ee6fd3b3a098b, State: Initialized, Role: FOLLOWER
I20260928 16:48:36.865403 15163 ts_tablet_manager.cc:1428] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260928 16:48:36.865777 15162 consensus_queue.cc:260] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:36.866993 15144 heartbeater.cc:503] Master 127.26.101.62:38761 was elected leader, sending a full tablet report...
I20260928 16:48:36.867581 15162 ts_tablet_manager.cc:1428] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
W20260928 16:48:36.998306 15145 tablet.cc:2369] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:48:37.026119 15014 tablet.cc:2369] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:37.033099 15169 raft_consensus.cc:482] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:48:37.033421 15169 raft_consensus.cc:504] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:37.035262 15169 leader_election.cc:288] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122c11fc6c894181a53cde61b730e9cb (127.26.101.2:38845), 8ec3e36f36274f6ca48ee6fd3b3a098b (127.26.101.3:37675)
I20260928 16:48:37.042572 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.035743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:34301 (local address 127.26.101.3:37675)
0928 16:48:37.035948 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.035955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.036871 (+ 916us) server_negotiation.cc:407] Connection header received
0928 16:48:37.037034 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.037037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.037102 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.037191 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.037901 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.038637 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.040803 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.041645 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.041814 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.041889 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.042194 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.042324 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:48:37.042596 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.035898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.26.101.1:56649 (local address 127.26.101.2:38845)
0928 16:48:37.036113 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.036118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.036766 (+ 648us) server_negotiation.cc:407] Connection header received
0928 16:48:37.036939 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.036945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.037045 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.037117 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:48:37.037827 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.038578 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.040949 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.041712 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.041876 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.041948 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.042219 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.042346 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:48:37.042693 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.035752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.2:38845 (local address 127.26.101.1:56649)
0928 16:48:37.036628 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.036649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.036795 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.037158 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.037163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.037227 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.037649 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.037660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.038727 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.038732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.040741 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.040759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.041922 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.041929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.042056 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.042079 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.042164 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.042308 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":112,"thread_start_us":498,"threads_started":1}
I20260928 16:48:37.042722 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.035851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.3:37675 (local address 127.26.101.1:34301)
0928 16:48:37.036731 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 16:48:37.036751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:48:37.036908 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:48:37.037227 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:48:37.037232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:48:37.037256 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:48:37.037727 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.037740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.038796 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.038799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.040593 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:48:37.040610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.041868 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.041875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:48:37.042033 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.042059 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:48:37.042135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:48:37.042278 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":108,"thread_start_us":498,"threads_started":1}
I20260928 16:48:37.043924 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfc4a7c01c140ccbf9b9fa5c01d2fb3" candidate_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" is_pre_election: true
I20260928 16:48:37.043918 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfc4a7c01c140ccbf9b9fa5c01d2fb3" candidate_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c11fc6c894181a53cde61b730e9cb" is_pre_election: true
I20260928 16:48:37.044413 15098 raft_consensus.cc:2454] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c630cd455f496fa7ad4eb06b5292fb in term 0.
I20260928 16:48:37.044413 14967 raft_consensus.cc:2454] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c630cd455f496fa7ad4eb06b5292fb in term 0.
I20260928 16:48:37.045084 14769 leader_election.cc:302] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122c11fc6c894181a53cde61b730e9cb, e2c630cd455f496fa7ad4eb06b5292fb; no voters:
I20260928 16:48:37.045488 15169 raft_consensus.cc:2781] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:48:37.045647 15169 raft_consensus.cc:482] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:48:37.045786 15169 raft_consensus.cc:3037] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.048419 15169 raft_consensus.cc:504] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:37.049085 15169 leader_election.cc:288] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [CANDIDATE]: Term 1 election: Requested vote from peers 122c11fc6c894181a53cde61b730e9cb (127.26.101.2:38845), 8ec3e36f36274f6ca48ee6fd3b3a098b (127.26.101.3:37675)
I20260928 16:48:37.049606 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfc4a7c01c140ccbf9b9fa5c01d2fb3" candidate_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122c11fc6c894181a53cde61b730e9cb"
I20260928 16:48:37.049643 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebfc4a7c01c140ccbf9b9fa5c01d2fb3" candidate_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b"
I20260928 16:48:37.049841 14967 raft_consensus.cc:3037] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.049839 15098 raft_consensus.cc:3037] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:37.052150 15098 raft_consensus.cc:2454] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c630cd455f496fa7ad4eb06b5292fb in term 1.
I20260928 16:48:37.052157 14967 raft_consensus.cc:2454] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c630cd455f496fa7ad4eb06b5292fb in term 1.
I20260928 16:48:37.052627 14768 leader_election.cc:302] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ec3e36f36274f6ca48ee6fd3b3a098b, e2c630cd455f496fa7ad4eb06b5292fb; no voters:
I20260928 16:48:37.052942 15169 raft_consensus.cc:2781] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:37.053493 15169 raft_consensus.cc:686] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 1 LEADER]: Becoming Leader. State: Replica: e2c630cd455f496fa7ad4eb06b5292fb, State: Running, Role: LEADER
I20260928 16:48:37.053973 15169 consensus_queue.cc:237] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } }
I20260928 16:48:37.058979 14693 catalog_manager.cc:5495] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c630cd455f496fa7ad4eb06b5292fb (127.26.101.1). New cstate: current_term: 1 leader_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c630cd455f496fa7ad4eb06b5292fb" member_type: VOTER last_known_addr { host: "127.26.101.1" port: 44191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 } health_report { overall_health: UNKNOWN } } }
W20260928 16:48:37.068936 14882 tablet.cc:2369] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:48:37.117255 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.111773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.26.101.1:44191)
0928 16:48:37.112009 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.112014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.112032 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:48:37.112257 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.112261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.112364 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.112502 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:48:37.112858 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.113561 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.115024 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.115482 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.115603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.116122 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:48:37.116646 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:48:37.116788 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.116895 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.116980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 16:48:37.126078 15098 raft_consensus.cc:1261] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 1 FOLLOWER]: Refusing update from remote peer e2c630cd455f496fa7ad4eb06b5292fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:37.126080 14967 raft_consensus.cc:1261] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 1 FOLLOWER]: Refusing update from remote peer e2c630cd455f496fa7ad4eb06b5292fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:48:37.127285 15173 consensus_queue.cc:1035] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "122c11fc6c894181a53cde61b730e9cb" member_type: VOTER last_known_addr { host: "127.26.101.2" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:37.127662 15169 consensus_queue.cc:1035] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec3e36f36274f6ca48ee6fd3b3a098b" member_type: VOTER last_known_addr { host: "127.26.101.3" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:48:37.139571 15176 mvcc.cc:205] Tried to move back new op lower bound from 7334355423737442304 to 7334355423456927744. Current Snapshot: MvccSnapshot[applied={T|T < 7334355423737442304}]
I20260928 16:48:37.154546 14692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:37.154830 14692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:48:37.159109 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef93c0e870ca4bc38b6c6195a49b514e: Sending DeleteTablet for 3 replicas of tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3
I20260928 16:48:37.160369 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='slave'} at 127.0.0.1:42194
I20260928 16:48:37.160405 14947 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='slave'} at 127.0.0.1:35058
I20260928 16:48:37.160815 15078 tablet_service.cc:1514] Processing DeleteTablet for tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:48:37 UTC) from {username='slave'} at 127.0.0.1:38516
I20260928 16:48:37.161224 15186 tablet_replica.cc:330] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: stopping tablet replica
I20260928 16:48:37.161686 15186 raft_consensus.cc:2229] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:48:37.161942 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14753
I20260928 16:48:37.162364 15186 raft_consensus.cc:2258] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:37.164742 15186 ts_tablet_manager.cc:1905] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:37.173147 15186 ts_tablet_manager.cc:1918] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:37.173451 15186 log.cc:1196] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-0/wal/wals/ebfc4a7c01c140ccbf9b9fa5c01d2fb3
I20260928 16:48:37.174027 15186 ts_tablet_manager.cc:1939] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P e2c630cd455f496fa7ad4eb06b5292fb: Deleting consensus metadata
I20260928 16:48:37.176688 14680 catalog_manager.cc:4841] TS e2c630cd455f496fa7ad4eb06b5292fb (127.26.101.1:44191): tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (table test-table [id=a661edc93da340628be1a35055259f67]) successfully deleted
I20260928 16:48:37.179687 15187 tablet_replica.cc:330] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: stopping tablet replica
I20260928 16:48:37.180200 15187 raft_consensus.cc:2229] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:37.180938 15187 raft_consensus.cc:2258] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:37.181529 15188 tablet_replica.cc:330] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: stopping tablet replica
I20260928 16:48:37.181923 15188 raft_consensus.cc:2229] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:48:37.182348 15188 raft_consensus.cc:2258] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:48:37.183336 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.164361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.26.101.1:44191)
0928 16:48:37.164521 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.164528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.173997 (+ 9469us) server_negotiation.cc:407] Connection header received
0928 16:48:37.174379 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.174384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.174504 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.174597 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.175293 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.176288 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.178966 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.179702 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.179829 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.181511 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.181524 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.181543 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:37.182378 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.182452 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:37.182482 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.182606 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.182795 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.182918 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:37.184650 14876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:37.185003 15187 ts_tablet_manager.cc:1905] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:37.186385 15188 ts_tablet_manager.cc:1905] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:48:37.192993 15187 ts_tablet_manager.cc:1918] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:37.193261 15187 log.cc:1196] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-1/wal/wals/ebfc4a7c01c140ccbf9b9fa5c01d2fb3
I20260928 16:48:37.193763 15188 ts_tablet_manager.cc:1918] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:48:37.193845 15187 ts_tablet_manager.cc:1939] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 122c11fc6c894181a53cde61b730e9cb: Deleting consensus metadata
I20260928 16:48:37.194020 15188 log.cc:1196] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Deleting WAL directory at /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613789516984-27028-0/minicluster-data/ts-2/wal/wals/ebfc4a7c01c140ccbf9b9fa5c01d2fb3
I20260928 16:48:37.194540 15188 ts_tablet_manager.cc:1939] T ebfc4a7c01c140ccbf9b9fa5c01d2fb3 P 8ec3e36f36274f6ca48ee6fd3b3a098b: Deleting consensus metadata
I20260928 16:48:37.196056 14678 catalog_manager.cc:4841] TS 122c11fc6c894181a53cde61b730e9cb (127.26.101.2:38845): tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (table test-table [id=a661edc93da340628be1a35055259f67]) successfully deleted
I20260928 16:48:37.196306 14677 catalog_manager.cc:4841] TS 8ec3e36f36274f6ca48ee6fd3b3a098b (127.26.101.3:37675): tablet ebfc4a7c01c140ccbf9b9fa5c01d2fb3 (table test-table [id=a661edc93da340628be1a35055259f67]) successfully deleted
I20260928 16:48:37.286132 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14753
I20260928 16:48:37.307464 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14885
I20260928 16:48:37.314827 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.308326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.26.101.2:38845)
0928 16:48:37.308497 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.308501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.308520 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:48:37.308624 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.308628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.308720 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.308805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.309226 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.310071 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.311537 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.311996 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.312110 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.313339 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.313350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.313365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:37.314019 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.314071 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:37.314094 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.314204 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.314371 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.314494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:48:37.315524 15008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:37.405220 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14885
I20260928 16:48:37.421068 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15016
I20260928 16:48:37.427986 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:37.421942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.26.101.3:37675)
0928 16:48:37.422133 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:37.422139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:37.422159 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:48:37.422303 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:37.422309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:37.422414 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:37.422501 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:37.422899 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.423690 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.425018 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:37.425415 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:37.425526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:37.426629 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:37.426639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:37.426654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:37.427270 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:37.427319 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:37.427346 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:37.427444 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:37.427570 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:48:37.427680 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:37.428678 15139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:37.515981 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15016
I20260928 16:48:37.531147 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 14662
I20260928 16:48:37.532199 14724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:37.643201 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 14662
2026-09-28T16:48:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2871 ms)
[----------] 6 tests from EncryptionPolicyTest (21982 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:37 dist-test-slave-t0tw krb5kdc[15196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:37 dist-test-slave-t0tw krb5kdc[15196](info): set up 2 sockets
Sep 28 16:48:37 dist-test-slave-t0tw krb5kdc[15196](info): commencing operation
krb5kdc: starting...
W20260928 16:48:38.691299 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:38 dist-test-slave-t0tw krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:38.963130 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:39.303968 15209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:39.304291 15209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:39.304417 15209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:39.311913 15209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:39.312042 15209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:39.312096 15209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:39.312178 15209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:39.312245 15209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:39.322587 15209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:39.323994 15209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:39.325352 15209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:39.337360 15214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:39.337317 15217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:39.337383 15215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:39.338368 15209 server_base.cc:1034] running on GCE node
I20260928 16:48:39.339229 15209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:39.340723 15209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:39.341933 15209 hybrid_clock.cc:648] HybridClock initialized: now 1790614119341884 us; error 46 us; skew 500 ppm
I20260928 16:48:39.342308 15209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:39 dist-test-slave-t0tw krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:39.345628 15209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:39.347276 15209 webserver.cc:460] Webserver started at http://127.0.0.1:41109/ using document root <none> and password file <none>
I20260928 16:48:39.347990 15209 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:39.348110 15209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:39.348409 15209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:39.351554 15209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "40e4ad800c1c47029a1d03b89c71c151"
format_stamp: "Formatted at 2026-09-28 16:48:39 on dist-test-slave-t0tw"
server_key: "ec57d3ff8921131c6dc349da40b2ea90"
server_key_iv: "4818d9b7f49744e3d32e2b10c199ee2f"
server_key_version: "encryptionkey@0"
I20260928 16:48:39.352325 15209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "40e4ad800c1c47029a1d03b89c71c151"
format_stamp: "Formatted at 2026-09-28 16:48:39 on dist-test-slave-t0tw"
server_key: "ec57d3ff8921131c6dc349da40b2ea90"
server_key_iv: "4818d9b7f49744e3d32e2b10c199ee2f"
server_key_version: "encryptionkey@0"
I20260928 16:48:39.357769 15209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 16:48:39.361624 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:39.363149 15209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:48:39.363336 15209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "40e4ad800c1c47029a1d03b89c71c151"
format_stamp: "Formatted at 2026-09-28 16:48:39 on dist-test-slave-t0tw"
server_key: "ec57d3ff8921131c6dc349da40b2ea90"
server_key_iv: "4818d9b7f49744e3d32e2b10c199ee2f"
server_key_version: "encryptionkey@0"
I20260928 16:48:39.363541 15209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:39.384191 15209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:39.388687 15209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:39.388978 15209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:39.406360 15209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37097
I20260928 16:48:39.406391 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37097 every 8 connection(s)
I20260928 16:48:39.407896 15209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:39.408492 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15209
I20260928 16:48:39.408679 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:39.409459 27028 external_mini_cluster.cc:1428] Setting key c67df9d5a30b393647e963f06a98c0ba
I20260928 16:48:39.412305 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:48:39 dist-test-slave-t0tw krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:39.425961 15277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151: Bootstrap starting.
I20260928 16:48:39.429523 15277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:39.430830 15277 log.cc:824] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:39.433427 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:39.411500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48525 (local address 127.0.0.1:37097)
0928 16:48:39.412261 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:39.412273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:39.412320 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:48:39.413981 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:39.414004 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:39.414399 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:39.414754 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:39.416636 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:39.418069 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:39.419964 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:39.420717 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:39.420863 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:39.425931 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:39.425940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:39.425960 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:39.426009 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:39.429043 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:39.430027 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:39.430035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:39.430041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:39.430220 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:39.430711 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:39.430714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:39.430716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:39.431362 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:39.431730 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:39.432160 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 16:48:39.432460 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":333,"threads_started":1}
I20260928 16:48:39.433939 15277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151: No bootstrap required, opened a new log
I20260928 16:48:39.437235 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:39.441674 15277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260928 16:48:39.442293 15277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:39.442400 15277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e4ad800c1c47029a1d03b89c71c151, State: Initialized, Role: FOLLOWER
I20260928 16:48:39.443039 15277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260928 16:48:39.443326 15277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:39.443456 15277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:39.443612 15277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:39.446378 15277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260928 16:48:39.446950 15277 leader_election.cc:302] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e4ad800c1c47029a1d03b89c71c151; no voters:
I20260928 16:48:39.447680 15277 leader_election.cc:288] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:39.447830 15287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:39.449823 15277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:39.450845 15287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [term 1 LEADER]: Becoming Leader. State: Replica: 40e4ad800c1c47029a1d03b89c71c151, State: Running, Role: LEADER
I20260928 16:48:39.451478 15287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260928 16:48:39.454476 15288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e4ad800c1c47029a1d03b89c71c151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260928 16:48:39.454881 15288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:39.459786 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:39.442029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47127 (local address 127.0.0.1:37097)
0928 16:48:39.442208 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:39.442213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:39.442539 (+ 326us) server_negotiation.cc:407] Connection header received
0928 16:48:39.442703 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:39.442708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:39.442790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:39.442888 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:39.444383 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:39.445518 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:39.446716 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:39.447438 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:39.447581 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:39.450809 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:39.450821 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:39.450830 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:39.450875 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:39.454833 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:39.456108 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:39.456115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:39.456119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:39.456316 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:39.456963 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:39.456969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:39.456973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:39.457427 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:39.457736 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:39.459140 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0928 16:48:39.459389 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:48:39.459738 15289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e4ad800c1c47029a1d03b89c71c151. Latest consensus state: current_term: 1 leader_uuid: "40e4ad800c1c47029a1d03b89c71c151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e4ad800c1c47029a1d03b89c71c151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260928 16:48:39.462317 15289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:39.462329 15295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:39.467959 15295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:39.476647 15295 catalog_manager.cc:1347] Generated new cluster ID: 1aa4408a879842f98653a8ffe66fb4bf
I20260928 16:48:39.476779 15295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:39.495769 15295 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:39.497771 15295 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:39.518685 15295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e4ad800c1c47029a1d03b89c71c151: Generated new TSK 0
I20260928 16:48:39.519925 15295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:39.538166 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15209
I20260928 16:48:39.539515 15271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:39.634994 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15209
W20260928 16:48:39.651500 15194 connection.cc:535] client connection to 127.0.0.1:37097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37097): Connection reset by peer (error 104)
2026-09-28T16:48:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:39 dist-test-slave-t0tw krb5kdc[15309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:39 dist-test-slave-t0tw krb5kdc[15309](info): set up 2 sockets
Sep 28 16:48:39 dist-test-slave-t0tw krb5kdc[15309](info): commencing operation
krb5kdc: starting...
W20260928 16:48:40.662997 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:40 dist-test-slave-t0tw krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:40.930905 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:41.279332 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:41.279665 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:41.279804 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:41.287374 15322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:41.287505 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:41.287582 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:41.287668 15322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:41.287726 15322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:41.297997 15322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:41.299449 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:41.300796 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:41.312476 15327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:41.312541 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:41.312672 15330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:41.313373 15322 server_base.cc:1034] running on GCE node
I20260928 16:48:41.314224 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:41.315814 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:41.317021 15322 hybrid_clock.cc:648] HybridClock initialized: now 1790614121316974 us; error 45 us; skew 500 ppm
I20260928 16:48:41.317363 15322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:41 dist-test-slave-t0tw krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:41.320601 15322 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:41.322072 15322 webserver.cc:460] Webserver started at http://127.0.0.1:45583/ using document root <none> and password file <none>
I20260928 16:48:41.322782 15322 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:41.322906 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:41.323243 15322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:41.326371 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "667ed85962f74151806087ac6f450870"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-t0tw"
server_key: "751aa218472ca0e100b5532cab64dda9"
server_key_iv: "31246dbe3ab30b40850ee85e08eb662d"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.327242 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "667ed85962f74151806087ac6f450870"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-t0tw"
server_key: "751aa218472ca0e100b5532cab64dda9"
server_key_iv: "31246dbe3ab30b40850ee85e08eb662d"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.332661 15322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 16:48:41.336532 15337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:41.338035 15322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:41.338234 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "667ed85962f74151806087ac6f450870"
format_stamp: "Formatted at 2026-09-28 16:48:41 on dist-test-slave-t0tw"
server_key: "751aa218472ca0e100b5532cab64dda9"
server_key_iv: "31246dbe3ab30b40850ee85e08eb662d"
server_key_version: "encryptionkey@0"
I20260928 16:48:41.338446 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:41.368322 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:41.372723 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:41.372997 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:41.390607 15322 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42787
I20260928 16:48:41.390630 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42787 every 8 connection(s)
I20260928 16:48:41.392031 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:41.395913 15389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:41.396200 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15322
I20260928 16:48:41.396371 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:41.396817 27028 external_mini_cluster.cc:1428] Setting key 5f3088326d068acb2a9f7906814ef783
I20260928 16:48:41.411984 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:41.398682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50225 (local address 127.0.0.1:42787)
0928 16:48:41.399606 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:41.399622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:41.399674 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:48:41.401340 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:41.401373 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:41.401958 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:41.402460 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:41.403050 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.405103 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.406532 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.407645 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.407859 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:41.409087 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:41.409098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:41.409118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:41.409908 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:41.410010 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:41.410043 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:41.410204 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:41.410421 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 16:48:41.410617 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":395,"threads_started":1}
I20260928 16:48:41.411984 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870: Bootstrap starting.
I20260928 16:48:41.415498 15389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:41.415854 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:41.416618 15389 log.cc:824] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:41.419809 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870: No bootstrap required, opened a new log
I20260928 16:48:41.427699 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:41.420600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34533 (local address 127.0.0.1:42787)
0928 16:48:41.420810 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:41.420817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:41.421174 (+ 357us) server_negotiation.cc:407] Connection header received
0928 16:48:41.421389 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:41.421393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:41.421454 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:41.421526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:41.422098 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.422980 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.424974 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:41.425500 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:41.425602 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:41.426850 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:41.426855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:41.426858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:41.426878 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:41.426922 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:41.426937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:41.427028 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:41.427269 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 16:48:41.427317 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:48:41.429704 15389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260928 16:48:41.430099 15389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:41.430194 15389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667ed85962f74151806087ac6f450870, State: Initialized, Role: FOLLOWER
I20260928 16:48:41.430758 15389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260928 16:48:41.430995 15389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:41.431167 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:41.431334 15389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:41.433599 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260928 16:48:41.434239 15389 leader_election.cc:302] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667ed85962f74151806087ac6f450870; no voters:
I20260928 16:48:41.434880 15389 leader_election.cc:288] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:41.434962 15399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:41.435658 15399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [term 1 LEADER]: Becoming Leader. State: Replica: 667ed85962f74151806087ac6f450870, State: Running, Role: LEADER
I20260928 16:48:41.436223 15399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260928 16:48:41.436988 15389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:41.439291 15401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667ed85962f74151806087ac6f450870. Latest consensus state: current_term: 1 leader_uuid: "667ed85962f74151806087ac6f450870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260928 16:48:41.439321 15400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667ed85962f74151806087ac6f450870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667ed85962f74151806087ac6f450870" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260928 16:48:41.439756 15400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:41.439755 15401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:41.444295 15405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:41.450021 15405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:41.460386 15405 catalog_manager.cc:1347] Generated new cluster ID: badc68d7b5d84cc5866472640c8cc586
I20260928 16:48:41.460529 15405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:41.491004 15405 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:41.492393 15405 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:41.502490 15405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667ed85962f74151806087ac6f450870: Generated new TSK 0
I20260928 16:48:41.503480 15405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:41.517310 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15322
W20260928 16:48:41.518572 15384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50225
W20260928 16:48:41.519385 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:41.519564 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15322
W20260928 16:48:41.535806 15307 connection.cc:535] client connection to 127.0.0.1:42787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42787): Connection reset by peer (error 104)
2026-09-28T16:48:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:41 dist-test-slave-t0tw krb5kdc[15420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:41 dist-test-slave-t0tw krb5kdc[15420](info): set up 2 sockets
Sep 28 16:48:41 dist-test-slave-t0tw krb5kdc[15420](info): commencing operation
krb5kdc: starting...
W20260928 16:48:42.550887 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:42 dist-test-slave-t0tw krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:42.813858 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:43.149546 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:43.149849 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:43.149983 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:43.157536 15433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:43.157666 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:43.157727 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:43.157811 15433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:43.157871 15433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:43.168515 15433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:43.169873 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:43.171242 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:43.183300 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:43.183410 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:43.183588 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:43.184234 15433 server_base.cc:1034] running on GCE node
I20260928 16:48:43.185092 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:43.186623 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:43.187855 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790614123187788 us; error 60 us; skew 500 ppm
I20260928 16:48:43.188297 15433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:43 dist-test-slave-t0tw krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:43.198675 15433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:43.200527 15433 webserver.cc:460] Webserver started at http://127.0.0.1:35513/ using document root <none> and password file <none>
I20260928 16:48:43.201270 15433 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:43.201383 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:43.201696 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:43.204909 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "d82aab41191a48eca7a698437baf7b03"
format_stamp: "Formatted at 2026-09-28 16:48:43 on dist-test-slave-t0tw"
server_key: "89633617d6e8651b497ea3495f8b13f0"
server_key_iv: "2863bde37d655282a997e8115d92c5df"
server_key_version: "encryptionkey@0"
I20260928 16:48:43.205746 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "d82aab41191a48eca7a698437baf7b03"
format_stamp: "Formatted at 2026-09-28 16:48:43 on dist-test-slave-t0tw"
server_key: "89633617d6e8651b497ea3495f8b13f0"
server_key_iv: "2863bde37d655282a997e8115d92c5df"
server_key_version: "encryptionkey@0"
I20260928 16:48:43.211355 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:48:43.214968 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:43.216846 15433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:43.217044 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "d82aab41191a48eca7a698437baf7b03"
format_stamp: "Formatted at 2026-09-28 16:48:43 on dist-test-slave-t0tw"
server_key: "89633617d6e8651b497ea3495f8b13f0"
server_key_iv: "2863bde37d655282a997e8115d92c5df"
server_key_version: "encryptionkey@0"
I20260928 16:48:43.217278 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:43.223006 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:43.227699 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:43.227988 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:43.245265 15433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40771
I20260928 16:48:43.245311 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40771 every 8 connection(s)
I20260928 16:48:43.246709 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:43.249320 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15433
I20260928 16:48:43.249478 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:43.249899 27028 external_mini_cluster.cc:1428] Setting key a3491c3dfcc24f316354896375a139da
I20260928 16:48:43.250836 15500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:43.257737 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:43.251638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59571 (local address 127.0.0.1:40771)
0928 16:48:43.252430 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:43.252441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:43.252483 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:48:43.254072 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:43.254096 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:43.254463 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:43.254845 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:43.256043 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:43.256049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:43.256119 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:43.256146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:43.256227 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:43.256249 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:43.256441 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:43.256537 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 16:48:43.256736 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":357,"threads_started":1}
I20260928 16:48:43.261437 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:43.264817 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03: Bootstrap starting.
I20260928 16:48:43.268752 15500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:43.269227 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:43.266748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40815 (local address 127.0.0.1:40771)
0928 16:48:43.266946 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:43.266953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:43.267378 (+ 425us) server_negotiation.cc:407] Connection header received
0928 16:48:43.267518 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:43.267522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:43.267588 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:43.267711 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:43.268558 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:43.268562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:43.268567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:43.268585 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:43.268649 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:43.268671 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:43.268795 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:43.268866 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 16:48:43.268927 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:48:43.270053 15500 log.cc:824] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:43.272892 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03: No bootstrap required, opened a new log
I20260928 16:48:43.280287 15500 raft_consensus.cc:348] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } }
I20260928 16:48:43.280709 15500 raft_consensus.cc:374] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:43.280804 15500 raft_consensus.cc:729] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82aab41191a48eca7a698437baf7b03, State: Initialized, Role: FOLLOWER
I20260928 16:48:43.281380 15500 consensus_queue.cc:260] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } }
I20260928 16:48:43.281605 15500 raft_consensus.cc:388] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:43.281737 15500 raft_consensus.cc:482] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:43.281900 15500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:43.284195 15500 raft_consensus.cc:504] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } }
I20260928 16:48:43.284703 15500 leader_election.cc:302] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d82aab41191a48eca7a698437baf7b03; no voters:
I20260928 16:48:43.285257 15500 leader_election.cc:288] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:43.285398 15510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:43.286034 15510 raft_consensus.cc:686] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [term 1 LEADER]: Becoming Leader. State: Replica: d82aab41191a48eca7a698437baf7b03, State: Running, Role: LEADER
I20260928 16:48:43.286593 15510 consensus_queue.cc:237] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } }
I20260928 16:48:43.287393 15500 sys_catalog.cc:564] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:43.289691 15512 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d82aab41191a48eca7a698437baf7b03. Latest consensus state: current_term: 1 leader_uuid: "d82aab41191a48eca7a698437baf7b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } } }
I20260928 16:48:43.290102 15512 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:43.290699 15511 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d82aab41191a48eca7a698437baf7b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82aab41191a48eca7a698437baf7b03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40771 } } }
I20260928 16:48:43.291103 15511 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:43.292259 15515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:43.297401 15515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:43.306056 15515 catalog_manager.cc:1347] Generated new cluster ID: 6281948a85664c81978985d02a50b32c
I20260928 16:48:43.306185 15515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:43.353947 15515 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:43.354403 15515 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:43.364355 15515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d82aab41191a48eca7a698437baf7b03: Generated new TSK 0
I20260928 16:48:43.365253 15515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:43.380975 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15433
W20260928 16:48:43.382431 15495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59571
W20260928 16:48:43.383232 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15433: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:43.383374 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15433
2026-09-28T16:48:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:43 dist-test-slave-t0tw krb5kdc[15531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:43 dist-test-slave-t0tw krb5kdc[15531](info): set up 2 sockets
Sep 28 16:48:43 dist-test-slave-t0tw krb5kdc[15531](info): commencing operation
krb5kdc: starting...
W20260928 16:48:44.419176 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:44 dist-test-slave-t0tw krb5kdc[15531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:44.686369 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:45.018567 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:45.018872 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:45.018967 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:45.026319 15544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:45.026446 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:45.026495 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:45.026556 15544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:45.026599 15544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:45.036767 15544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:45.038079 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:45.039428 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:45.051554 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:45.051474 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:45.051537 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:45.052435 15544 server_base.cc:1034] running on GCE node
I20260928 16:48:45.053411 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:45.054958 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:45.056202 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790614125056155 us; error 37 us; skew 500 ppm
I20260928 16:48:45.056577 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:45 dist-test-slave-t0tw krb5kdc[15531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:45.060094 15544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:45.061833 15544 webserver.cc:460] Webserver started at http://127.0.0.1:45187/ using document root <none> and password file <none>
I20260928 16:48:45.062546 15544 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:45.062680 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:45.062975 15544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:45.066196 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "8449d429db1045fea43682311fa2ec15"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-t0tw"
server_key: "9d08b16e842580c59ab710e4bfa2a8f3"
server_key_iv: "ea047da7a645eb46a75899de31195890"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.067004 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "8449d429db1045fea43682311fa2ec15"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-t0tw"
server_key: "9d08b16e842580c59ab710e4bfa2a8f3"
server_key_iv: "ea047da7a645eb46a75899de31195890"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.072631 15544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 16:48:45.076467 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:45.077929 15544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:45.078189 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8449d429db1045fea43682311fa2ec15"
format_stamp: "Formatted at 2026-09-28 16:48:45 on dist-test-slave-t0tw"
server_key: "9d08b16e842580c59ab710e4bfa2a8f3"
server_key_iv: "ea047da7a645eb46a75899de31195890"
server_key_version: "encryptionkey@0"
I20260928 16:48:45.078389 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:45.102612 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:45.107026 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:45.107388 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:45.125434 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41243
I20260928 16:48:45.125473 15610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41243 every 8 connection(s)
I20260928 16:48:45.126802 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:45.130661 15611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:45.131891 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15544
I20260928 16:48:45.132134 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:45.132593 27028 external_mini_cluster.cc:1428] Setting key b7229b44ae0faaefb09d3ace958882d9
I20260928 16:48:45.144091 15611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15: Bootstrap starting.
I20260928 16:48:45.147805 15611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15: Neither blocks nor log segments found. Creating new log.
Sep 28 16:48:45 dist-test-slave-t0tw krb5kdc[15531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:45.149027 15611 log.cc:824] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:45.152243 15611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15: No bootstrap required, opened a new log
I20260928 16:48:45.157717 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:45.134619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45939 (local address 127.0.0.1:41243)
0928 16:48:45.135658 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:45.135679 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:45.136073 (+ 394us) server_negotiation.cc:407] Connection header received
0928 16:48:45.138489 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:45.138523 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:45.139158 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:45.139641 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:45.141004 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.143057 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.144424 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.145429 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.145617 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:45.150377 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:45.150386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:45.150402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:45.150441 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:45.153440 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:45.154197 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:45.154204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:45.154210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:45.154388 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:45.154788 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:45.154792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:45.154794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:45.155417 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:45.155682 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:45.156130 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 16:48:45.156474 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":479,"threads_started":1}
I20260928 16:48:45.161056 15611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } }
I20260928 16:48:45.161588 15611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:45.161715 15611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8449d429db1045fea43682311fa2ec15, State: Initialized, Role: FOLLOWER
I20260928 16:48:45.161963 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:45.162351 15611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } }
I20260928 16:48:45.162627 15611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:45.162765 15611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:45.162921 15611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:45.166145 15611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } }
I20260928 16:48:45.166764 15611 leader_election.cc:302] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8449d429db1045fea43682311fa2ec15; no voters:
I20260928 16:48:45.167484 15611 leader_election.cc:288] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:45.167649 15620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:45.168273 15620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [term 1 LEADER]: Becoming Leader. State: Replica: 8449d429db1045fea43682311fa2ec15, State: Running, Role: LEADER
I20260928 16:48:45.168941 15620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } }
I20260928 16:48:45.171173 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8449d429db1045fea43682311fa2ec15. Latest consensus state: current_term: 1 leader_uuid: "8449d429db1045fea43682311fa2ec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } } }
I20260928 16:48:45.171473 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:45.171963 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8449d429db1045fea43682311fa2ec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8449d429db1045fea43682311fa2ec15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41243 } } }
I20260928 16:48:45.172319 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:45.176074 15611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:45.180639 15625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:45.188133 15625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:45.193729 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:45.167285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54203 (local address 127.0.0.1:41243)
0928 16:48:45.167480 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:45.167486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:45.174009 (+ 6523us) server_negotiation.cc:407] Connection header received
0928 16:48:45.174252 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:45.174259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:45.174354 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:45.174468 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:45.176880 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.178215 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.180240 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:45.181030 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:45.181179 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:45.184551 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:45.184558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:45.184563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:45.184605 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:45.189027 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:45.190177 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:45.190182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:45.190187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:45.190387 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:45.190928 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:45.190935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:45.190939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:45.191462 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:45.192867 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:45.193057 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 16:48:45.193315 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:48:45.206329 15625 catalog_manager.cc:1347] Generated new cluster ID: 5c9c07aec0be45a3a351039b4d284365
I20260928 16:48:45.206470 15625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:45.229918 15625 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:45.232077 15625 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:45.247931 15625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8449d429db1045fea43682311fa2ec15: Generated new TSK 0
I20260928 16:48:45.249135 15625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:45.264038 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15544
I20260928 16:48:45.265237 15606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:45.359890 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15544
2026-09-28T16:48:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:45 dist-test-slave-t0tw krb5kdc[15643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:45 dist-test-slave-t0tw krb5kdc[15643](info): set up 2 sockets
Sep 28 16:48:45 dist-test-slave-t0tw krb5kdc[15643](info): commencing operation
krb5kdc: starting...
W20260928 16:48:46.387008 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:46 dist-test-slave-t0tw krb5kdc[15643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:46.651028 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:46.986825 15656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:46.987190 15656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:46.987339 15656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:46.994743 15656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:46.994868 15656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:46.994927 15656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:46.995004 15656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:46.995065 15656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:47.005167 15656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:47.006492 15656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:47.007843 15656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:47.019651 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:47.019683 15661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:47.019719 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:47.020334 15656 server_base.cc:1034] running on GCE node
I20260928 16:48:47.021097 15656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:47.022533 15656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:47.023729 15656 hybrid_clock.cc:648] HybridClock initialized: now 1790614127023680 us; error 46 us; skew 500 ppm
I20260928 16:48:47.024065 15656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:47 dist-test-slave-t0tw krb5kdc[15643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:47.027258 15656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:47.028779 15656 webserver.cc:460] Webserver started at http://127.0.0.1:38133/ using document root <none> and password file <none>
I20260928 16:48:47.029462 15656 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:47.029583 15656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:47.029881 15656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:47.032961 15656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "5d8bd1edb98c402ebaf104fd362ace36"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-t0tw"
server_key: "3e7f81dff5872744bfdb887b8969b79b"
server_key_iv: "d111a23d234017f8590783373858cef9"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.033756 15656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "5d8bd1edb98c402ebaf104fd362ace36"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-t0tw"
server_key: "3e7f81dff5872744bfdb887b8969b79b"
server_key_iv: "d111a23d234017f8590783373858cef9"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.039348 15656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:48:47.043191 15671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:47.044694 15656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:47.044880 15656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "5d8bd1edb98c402ebaf104fd362ace36"
format_stamp: "Formatted at 2026-09-28 16:48:47 on dist-test-slave-t0tw"
server_key: "3e7f81dff5872744bfdb887b8969b79b"
server_key_iv: "d111a23d234017f8590783373858cef9"
server_key_version: "encryptionkey@0"
I20260928 16:48:47.045087 15656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:47.081104 15656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:47.085702 15656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:47.085989 15656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:47.104427 15656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33045
I20260928 16:48:47.104447 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33045 every 8 connection(s)
I20260928 16:48:47.105863 15656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:47.107493 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15656
I20260928 16:48:47.107642 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:47.108089 27028 external_mini_cluster.cc:1428] Setting key 1455abf5dfad0d6e95f1a251a3439db1
I20260928 16:48:47.110139 15723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:47.124342 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:47.110109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38707 (local address 127.0.0.1:33045)
0928 16:48:47.110965 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:47.110981 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:47.111048 (+ 67us) server_negotiation.cc:407] Connection header received
0928 16:48:47.113243 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:47.113274 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:47.113852 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:47.114382 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:47.114959 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:47.117178 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:47.118761 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:47.119861 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:47.120076 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:47.121313 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:47.121324 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:47.121343 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:47.122153 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:47.122260 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:47.122295 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:47.122406 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:47.122634 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:48:47.122956 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":364,"threads_started":1}
I20260928 16:48:47.125216 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36: Bootstrap starting.
I20260928 16:48:47.128139 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:47.128531 15723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:47.129756 15723 log.cc:824] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:47.133144 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36: No bootstrap required, opened a new log
I20260928 16:48:47.140059 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:47.133109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34433 (local address 127.0.0.1:33045)
0928 16:48:47.133337 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:47.133343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:47.133723 (+ 380us) server_negotiation.cc:407] Connection header received
0928 16:48:47.133940 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:47.133944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:47.134009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:47.134146 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:47.134707 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:47.135568 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:47.137453 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:47.137914 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:47.138038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:47.139259 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:47.139264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:47.139267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:47.139287 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:47.139331 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:47.139347 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:47.139441 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:47.139620 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 16:48:47.139713 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:48:47.142755 15723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260928 16:48:47.143292 15723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:47.143414 15723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8bd1edb98c402ebaf104fd362ace36, State: Initialized, Role: FOLLOWER
I20260928 16:48:47.144038 15723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260928 16:48:47.144279 15723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:47.144403 15723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:47.144563 15723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:47.146731 15723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260928 16:48:47.147271 15723 leader_election.cc:302] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d8bd1edb98c402ebaf104fd362ace36; no voters:
I20260928 16:48:47.147867 15723 leader_election.cc:288] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:47.148030 15733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:47.149032 15733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [term 1 LEADER]: Becoming Leader. State: Replica: 5d8bd1edb98c402ebaf104fd362ace36, State: Running, Role: LEADER
I20260928 16:48:47.149652 15733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } }
I20260928 16:48:47.150087 15723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:47.152447 15734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d8bd1edb98c402ebaf104fd362ace36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } } }
I20260928 16:48:47.152477 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8bd1edb98c402ebaf104fd362ace36. Latest consensus state: current_term: 1 leader_uuid: "5d8bd1edb98c402ebaf104fd362ace36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8bd1edb98c402ebaf104fd362ace36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33045 } } }
I20260928 16:48:47.152882 15734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:47.152930 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:47.154341 15739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:47.162147 15739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:47.171725 15739 catalog_manager.cc:1347] Generated new cluster ID: c26fe27ca8614e1baec64a91501e27b4
I20260928 16:48:47.171849 15739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:47.186043 15739 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:47.187511 15739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:47.201129 15739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d8bd1edb98c402ebaf104fd362ace36: Generated new TSK 0
I20260928 16:48:47.202033 15739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:47.209688 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15656
W20260928 16:48:47.210893 15718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38707
W20260928 16:48:47.211623 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:47.211786 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15656
2026-09-28T16:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:47 dist-test-slave-t0tw krb5kdc[15754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:47 dist-test-slave-t0tw krb5kdc[15754](info): set up 2 sockets
Sep 28 16:48:47 dist-test-slave-t0tw krb5kdc[15754](info): commencing operation
krb5kdc: starting...
W20260928 16:48:48.243191 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:48 dist-test-slave-t0tw krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:48.520237 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:48.870182 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:48.870533 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:48.870644 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:48.878283 15767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:48.878422 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:48.878475 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:48.878561 15767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:48.878618 15767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:48.889602 15767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:48.891062 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:48.892678 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:48.904713 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:48.904740 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:48.904654 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:48.905763 15767 server_base.cc:1034] running on GCE node
I20260928 16:48:48.906621 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:48.908231 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:48.909490 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790614128909412 us; error 65 us; skew 500 ppm
I20260928 16:48:48.909875 15767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:48 dist-test-slave-t0tw krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:48.920410 15767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:48.922293 15767 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260928 16:48:48.923569 15767 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:48.923774 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:48.924166 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:48.927542 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "556c216364034e8388cb84681aace2fe"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-t0tw"
server_key: "e4c88787ea76663cf4d1b75b3007aeef"
server_key_iv: "a945c311630c4be671ac9846d8bc81df"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.928576 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "556c216364034e8388cb84681aace2fe"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-t0tw"
server_key: "e4c88787ea76663cf4d1b75b3007aeef"
server_key_iv: "a945c311630c4be671ac9846d8bc81df"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.934542 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 16:48:48.938642 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:48.940368 15767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:48.940589 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "556c216364034e8388cb84681aace2fe"
format_stamp: "Formatted at 2026-09-28 16:48:48 on dist-test-slave-t0tw"
server_key: "e4c88787ea76663cf4d1b75b3007aeef"
server_key_iv: "a945c311630c4be671ac9846d8bc81df"
server_key_version: "encryptionkey@0"
I20260928 16:48:48.940810 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:48.947181 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:48.952266 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:48.952556 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:48.972249 15767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44135
I20260928 16:48:48.972289 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44135 every 8 connection(s)
I20260928 16:48:48.973691 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:48.977463 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15767
I20260928 16:48:48.977649 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:48.977633 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:48.978175 27028 external_mini_cluster.cc:1428] Setting key cee2adadc05c4c16defb9d711a2d84c5
I20260928 16:48:48.987035 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:48.980152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50421 (local address 127.0.0.1:44135)
0928 16:48:48.981099 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:48.981112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:48.981164 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:48:48.982895 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:48.982924 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:48.983421 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:48.983840 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:48.985107 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:48.985115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:48.985131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:48.985151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:48.985230 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:48.985255 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:48.985479 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:48.985708 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 16:48:48.985993 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":438,"threads_started":1}
I20260928 16:48:48.992421 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe: Bootstrap starting.
I20260928 16:48:48.992479 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:48.996053 15834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:48.997468 15834 log.cc:824] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:48.999864 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:48.997196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42739 (local address 127.0.0.1:44135)
0928 16:48:48.997401 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:48.997407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:48.997717 (+ 310us) server_negotiation.cc:407] Connection header received
0928 16:48:48.997894 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:48.997900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:48.997985 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:48.998132 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:48.999188 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:48.999193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:48.999198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:48.999217 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:48.999264 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:48.999284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:48.999403 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:48.999522 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 16:48:48.999587 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:48:49.001219 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe: No bootstrap required, opened a new log
I20260928 16:48:49.009202 15834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260928 16:48:49.009644 15834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:49.009732 15834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c216364034e8388cb84681aace2fe, State: Initialized, Role: FOLLOWER
I20260928 16:48:49.010341 15834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260928 16:48:49.010659 15834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:49.010818 15834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:49.010993 15834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:49.013427 15834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260928 16:48:49.013989 15834 leader_election.cc:302] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556c216364034e8388cb84681aace2fe; no voters:
I20260928 16:48:49.014631 15834 leader_election.cc:288] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:49.014827 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:49.015549 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [term 1 LEADER]: Becoming Leader. State: Replica: 556c216364034e8388cb84681aace2fe, State: Running, Role: LEADER
I20260928 16:48:49.016105 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260928 16:48:49.016799 15834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:49.019045 15846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556c216364034e8388cb84681aace2fe. Latest consensus state: current_term: 1 leader_uuid: "556c216364034e8388cb84681aace2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260928 16:48:49.019516 15846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:49.019001 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556c216364034e8388cb84681aace2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c216364034e8388cb84681aace2fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260928 16:48:49.019711 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:49.020496 15850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:49.026100 15850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:49.036768 15850 catalog_manager.cc:1347] Generated new cluster ID: 4bb09e3de91a4b9a8f7f4e5f599185c7
I20260928 16:48:49.036913 15850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:49.053086 15850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:49.053660 15850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:49.065620 15850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556c216364034e8388cb84681aace2fe: Generated new TSK 0
I20260928 16:48:49.066639 15850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:49.085258 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15767
W20260928 16:48:49.086508 15829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50421
W20260928 16:48:49.087391 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15767: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:49.087574 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15767
2026-09-28T16:48:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:49 dist-test-slave-t0tw krb5kdc[15865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:49 dist-test-slave-t0tw krb5kdc[15865](info): set up 2 sockets
Sep 28 16:48:49 dist-test-slave-t0tw krb5kdc[15865](info): commencing operation
krb5kdc: starting...
W20260928 16:48:50.123055 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:50 dist-test-slave-t0tw krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:50.396720 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:50.729055 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:50.729393 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:50.729518 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:50.737011 15878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:50.737161 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:50.737208 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:50.737289 15878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:50.737347 15878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:50.747630 15878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:50.748999 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:50.750389 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:50.762387 15883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:50.762408 15886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:50.762835 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:50.764009 15878 server_base.cc:1034] running on GCE node
I20260928 16:48:50.764896 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:50.766505 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:50.767747 15878 hybrid_clock.cc:648] HybridClock initialized: now 1790614130767691 us; error 43 us; skew 500 ppm
I20260928 16:48:50.768131 15878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:50 dist-test-slave-t0tw krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:50.771533 15878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:50.773164 15878 webserver.cc:460] Webserver started at http://127.0.0.1:38953/ using document root <none> and password file <none>
I20260928 16:48:50.773905 15878 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:50.774047 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:50.774358 15878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:50.777876 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "ebdcae341af3425f88a47daabd1050ca"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-t0tw"
server_key: "0976673fc39c0f4b09b7b0311327c896"
server_key_iv: "9186fc07147bc03580ea4564c698bd59"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.778787 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "ebdcae341af3425f88a47daabd1050ca"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-t0tw"
server_key: "0976673fc39c0f4b09b7b0311327c896"
server_key_iv: "9186fc07147bc03580ea4564c698bd59"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.784466 15878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 16:48:50.788339 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:50.789755 15878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:48:50.789944 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "ebdcae341af3425f88a47daabd1050ca"
format_stamp: "Formatted at 2026-09-28 16:48:50 on dist-test-slave-t0tw"
server_key: "0976673fc39c0f4b09b7b0311327c896"
server_key_iv: "9186fc07147bc03580ea4564c698bd59"
server_key_version: "encryptionkey@0"
I20260928 16:48:50.790148 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:50.807829 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:50.812489 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:50.812786 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:50.831646 15878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35455
I20260928 16:48:50.831678 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35455 every 8 connection(s)
I20260928 16:48:50.833115 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:50.837004 15945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:50.843039 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15878
I20260928 16:48:50.843289 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:50.843744 27028 external_mini_cluster.cc:1428] Setting key 235c4d15e9b62561239d9a1b390de2bc
I20260928 16:48:50.849052 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca: Bootstrap starting.
I20260928 16:48:50.852459 15945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:50.853559 15945 log.cc:824] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:50.856694 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca: No bootstrap required, opened a new log
Sep 28 16:48:50 dist-test-slave-t0tw krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:50.866791 15945 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260928 16:48:50.867367 15945 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:50.867486 15945 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdcae341af3425f88a47daabd1050ca, State: Initialized, Role: FOLLOWER
I20260928 16:48:50.868220 15945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260928 16:48:50.868515 15945 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:50.868692 15945 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:50.868896 15945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:50.869643 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:50.845795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:33865 (local address 127.0.0.1:35455)
0928 16:48:50.846774 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:50.846791 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:50.846844 (+ 53us) server_negotiation.cc:407] Connection header received
0928 16:48:50.848720 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:50.848747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:50.849208 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:50.849678 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:50.851286 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.852627 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.853997 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.855140 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.855329 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:50.859954 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:50.859966 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:50.859993 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:50.860057 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:50.864382 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:50.865378 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:50.865388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:50.865398 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:50.865647 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:50.866145 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:50.866150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:50.866153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:50.866915 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:50.867275 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:50.867738 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 16:48:50.868090 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":471,"threads_started":1}
I20260928 16:48:50.871680 15945 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260928 16:48:50.872196 15945 leader_election.cc:302] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebdcae341af3425f88a47daabd1050ca; no voters:
I20260928 16:48:50.872869 15945 leader_election.cc:288] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:50.873035 15950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:50.873795 15950 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [term 1 LEADER]: Becoming Leader. State: Replica: ebdcae341af3425f88a47daabd1050ca, State: Running, Role: LEADER
I20260928 16:48:50.874390 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:50.874547 15950 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } }
I20260928 16:48:50.876132 15945 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:50.877379 15951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebdcae341af3425f88a47daabd1050ca. Latest consensus state: current_term: 1 leader_uuid: "ebdcae341af3425f88a47daabd1050ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } } }
I20260928 16:48:50.877786 15951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:50.883267 15952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebdcae341af3425f88a47daabd1050ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdcae341af3425f88a47daabd1050ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35455 } } }
I20260928 16:48:50.883610 15952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:50.887638 15956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:50.894078 15956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:50.906407 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:50.886715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:50471 (local address 127.0.0.1:35455)
0928 16:48:50.886928 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:50.886936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:50.887780 (+ 844us) server_negotiation.cc:407] Connection header received
0928 16:48:50.888013 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:50.888020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:50.888120 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:50.888248 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:50.890002 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.891231 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.893134 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:50.893809 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:50.893920 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:50.897687 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:50.897695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:50.897700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:50.897755 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:50.901745 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:50.902909 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:50.902915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:50.902919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:50.903103 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:50.903769 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:50.903774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:50.903778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:50.904283 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:50.904568 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:50.905802 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0928 16:48:50.906010 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:48:50.911926 15956 catalog_manager.cc:1347] Generated new cluster ID: 32f4b8bfb2524cb08611ea6e0bf8990c
I20260928 16:48:50.912061 15956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:50.939859 15956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:50.941285 15956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:50.954138 15956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebdcae341af3425f88a47daabd1050ca: Generated new TSK 0
I20260928 16:48:50.955227 15956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:50.962569 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15878
I20260928 16:48:50.963887 15940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:51.058845 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15878
W20260928 16:48:51.074873 15863 connection.cc:535] client connection to 127.0.0.1:35455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35455): Connection reset by peer (error 104)
2026-09-28T16:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:51 dist-test-slave-t0tw krb5kdc[15977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:51 dist-test-slave-t0tw krb5kdc[15977](info): set up 2 sockets
Sep 28 16:48:51 dist-test-slave-t0tw krb5kdc[15977](info): commencing operation
krb5kdc: starting...
W20260928 16:48:52.099303 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:52 dist-test-slave-t0tw krb5kdc[15977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:52.366974 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:52.700009 15990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:52.700340 15990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:52.700489 15990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:52.707917 15990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:52.708070 15990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:52.708153 15990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:52.708235 15990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:52.708302 15990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:52.718545 15990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:52.720013 15990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:52.721446 15990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:52.733551 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:52.733711 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:52.733500 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:52.734442 15990 server_base.cc:1034] running on GCE node
I20260928 16:48:52.735332 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:52.736768 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:52.737993 15990 hybrid_clock.cc:648] HybridClock initialized: now 1790614132737940 us; error 48 us; skew 500 ppm
I20260928 16:48:52.738358 15990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:52 dist-test-slave-t0tw krb5kdc[15977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:52.741595 15990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:52.743243 15990 webserver.cc:460] Webserver started at http://127.0.0.1:41909/ using document root <none> and password file <none>
I20260928 16:48:52.743942 15990 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:52.744072 15990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:52.744387 15990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:52.747505 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "a35b5c4ccd4a4fb99e069696270fc41d"
format_stamp: "Formatted at 2026-09-28 16:48:52 on dist-test-slave-t0tw"
server_key: "041f4dfde25bbcbe11e258395e887c14"
server_key_iv: "61bc14ac591dcb74407832e8ca415215"
server_key_version: "encryptionkey@0"
I20260928 16:48:52.748279 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "a35b5c4ccd4a4fb99e069696270fc41d"
format_stamp: "Formatted at 2026-09-28 16:48:52 on dist-test-slave-t0tw"
server_key: "041f4dfde25bbcbe11e258395e887c14"
server_key_iv: "61bc14ac591dcb74407832e8ca415215"
server_key_version: "encryptionkey@0"
I20260928 16:48:52.753619 15990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 16:48:52.757313 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:52.758713 15990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:48:52.758881 15990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "a35b5c4ccd4a4fb99e069696270fc41d"
format_stamp: "Formatted at 2026-09-28 16:48:52 on dist-test-slave-t0tw"
server_key: "041f4dfde25bbcbe11e258395e887c14"
server_key_iv: "61bc14ac591dcb74407832e8ca415215"
server_key_version: "encryptionkey@0"
I20260928 16:48:52.759116 15990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:52.771729 15990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:52.775709 15990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:52.775985 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:52.794255 15990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34085
I20260928 16:48:52.794243 16056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34085 every 8 connection(s)
I20260928 16:48:52.795809 15990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:52.799853 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:52.802634 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 15990
I20260928 16:48:52.802795 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:52.803280 27028 external_mini_cluster.cc:1428] Setting key 2e3567d7c87196943bc8721374a2563e
I20260928 16:48:52.811679 16057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d: Bootstrap starting.
I20260928 16:48:52.815728 16057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:52.817142 16057 log.cc:824] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:52.817214 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:52.805236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53359 (local address 127.0.0.1:34085)
0928 16:48:52.806045 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:52.806057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:52.806101 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:48:52.807848 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:52.807873 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:52.808308 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:52.808707 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:52.809281 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:52.810580 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:52.811938 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:52.812986 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:52.813183 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:52.814351 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:52.814363 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:52.814381 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:52.815121 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:52.815219 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:52.815256 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:52.815424 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:52.815663 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 16:48:52.815854 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":352,"threads_started":1}
I20260928 16:48:52.820674 16057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d: No bootstrap required, opened a new log
I20260928 16:48:52.821733 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:52.831226 16057 raft_consensus.cc:348] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } }
I20260928 16:48:52.831784 16057 raft_consensus.cc:374] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:52.831914 16057 raft_consensus.cc:729] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35b5c4ccd4a4fb99e069696270fc41d, State: Initialized, Role: FOLLOWER
I20260928 16:48:52.832587 16057 consensus_queue.cc:260] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } }
I20260928 16:48:52.832821 16057 raft_consensus.cc:388] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:52.832964 16057 raft_consensus.cc:482] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:52.833134 16057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:52.833935 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:52.827191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:44973 (local address 127.0.0.1:34085)
0928 16:48:52.827391 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:52.827397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:52.827619 (+ 222us) server_negotiation.cc:407] Connection header received
0928 16:48:52.827783 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:52.827789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:52.827875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:52.828005 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:52.828471 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:52.829717 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:52.831134 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:52.831772 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:52.831911 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:52.833110 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:52.833117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:52.833122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:52.833147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:52.833209 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:52.833231 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:52.833330 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:52.833556 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 16:48:52.833605 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 16:48:52.835776 16057 raft_consensus.cc:504] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } }
I20260928 16:48:52.836294 16057 leader_election.cc:302] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a35b5c4ccd4a4fb99e069696270fc41d; no voters:
I20260928 16:48:52.836834 16057 leader_election.cc:288] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:52.836969 16067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:52.837608 16067 raft_consensus.cc:686] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [term 1 LEADER]: Becoming Leader. State: Replica: a35b5c4ccd4a4fb99e069696270fc41d, State: Running, Role: LEADER
I20260928 16:48:52.838217 16067 consensus_queue.cc:237] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } }
I20260928 16:48:52.838886 16057 sys_catalog.cc:564] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:52.841486 16069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a35b5c4ccd4a4fb99e069696270fc41d. Latest consensus state: current_term: 1 leader_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } } }
I20260928 16:48:52.841794 16069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:52.843137 16068 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a35b5c4ccd4a4fb99e069696270fc41d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34085 } } }
I20260928 16:48:52.844192 16068 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:52.845427 16073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:52.850780 16073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:52.860134 16073 catalog_manager.cc:1347] Generated new cluster ID: 279c8196fc0f49e3bb6c69987b5059e7
I20260928 16:48:52.860261 16073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:52.879834 16073 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:52.881170 16073 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:52.896215 16073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a35b5c4ccd4a4fb99e069696270fc41d: Generated new TSK 0
I20260928 16:48:52.897396 16073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:52.917438 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15990
W20260928 16:48:52.918730 16052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:53359
W20260928 16:48:52.919697 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 15990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:52.919871 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 15990
W20260928 16:48:52.935786 15975 connection.cc:535] client connection to 127.0.0.1:34085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34085): Connection reset by peer (error 104)
2026-09-28T16:48:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:53 dist-test-slave-t0tw krb5kdc[16088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:53 dist-test-slave-t0tw krb5kdc[16088](info): set up 2 sockets
Sep 28 16:48:53 dist-test-slave-t0tw krb5kdc[16088](info): commencing operation
krb5kdc: starting...
W20260928 16:48:53.951134 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:54 dist-test-slave-t0tw krb5kdc[16088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:54.223455 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:54.557644 16101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:54.557988 16101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:54.558141 16101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:54.565635 16101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:54.565785 16101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:54.565869 16101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:54.565963 16101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:54.566020 16101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:54.576534 16101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:54.577857 16101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:54.579273 16101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:54.592672 16109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:54.592808 16106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:54.594074 16107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:54.596459 16101 server_base.cc:1034] running on GCE node
I20260928 16:48:54.597434 16101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:54.598996 16101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:54.600297 16101 hybrid_clock.cc:648] HybridClock initialized: now 1790614134600222 us; error 66 us; skew 500 ppm
I20260928 16:48:54.600689 16101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:54 dist-test-slave-t0tw krb5kdc[16088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:54.608740 16101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:54.610456 16101 webserver.cc:460] Webserver started at http://127.0.0.1:37859/ using document root <none> and password file <none>
I20260928 16:48:54.611312 16101 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:54.611452 16101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:54.611768 16101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:54.614912 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "db46737b068146259087da995f1bb4fa"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-t0tw"
server_key: "4b055825acf5d0671f8147f156756a67"
server_key_iv: "9be6a62452dc818bf8f3dd71834b3225"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.615784 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "db46737b068146259087da995f1bb4fa"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-t0tw"
server_key: "4b055825acf5d0671f8147f156756a67"
server_key_iv: "9be6a62452dc818bf8f3dd71834b3225"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.621093 16101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 16:48:54.624709 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:54.626092 16101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:48:54.626272 16101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "db46737b068146259087da995f1bb4fa"
format_stamp: "Formatted at 2026-09-28 16:48:54 on dist-test-slave-t0tw"
server_key: "4b055825acf5d0671f8147f156756a67"
server_key_iv: "9be6a62452dc818bf8f3dd71834b3225"
server_key_version: "encryptionkey@0"
I20260928 16:48:54.626472 16101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:54.632275 16101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:54.636375 16101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:54.636672 16101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:54.654865 16101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34545
I20260928 16:48:54.654897 16167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34545 every 8 connection(s)
I20260928 16:48:54.656307 16101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:54.658808 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16101
I20260928 16:48:54.659000 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:54.659556 27028 external_mini_cluster.cc:1428] Setting key 612f720f86dffa4d35ab6ddb7c5f404d
I20260928 16:48:54.660734 16168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:54.668545 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:54.661404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59545 (local address 127.0.0.1:34545)
0928 16:48:54.662270 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:54.662285 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:54.662340 (+ 55us) server_negotiation.cc:407] Connection header received
0928 16:48:54.664355 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:54.664388 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:54.664896 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:54.665404 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:54.666434 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:54.666443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:54.666462 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:54.666489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:54.666574 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:54.666603 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:54.666840 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:54.666985 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 16:48:54.667243 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
I20260928 16:48:54.672752 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:54.676357 16168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa: Bootstrap starting.
I20260928 16:48:54.680621 16168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:54.680936 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:54.678239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:43801 (local address 127.0.0.1:34545)
0928 16:48:54.678430 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:54.678436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:54.678954 (+ 518us) server_negotiation.cc:407] Connection header received
0928 16:48:54.679182 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:54.679186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:54.679240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:54.679323 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:54.680279 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:54.680284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:54.680287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:54.680302 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:54.680344 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:54.680359 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:54.680447 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:54.680607 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:48:54.680661 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:48:54.682010 16168 log.cc:824] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:54.684795 16168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa: No bootstrap required, opened a new log
I20260928 16:48:54.692430 16168 raft_consensus.cc:348] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260928 16:48:54.692848 16168 raft_consensus.cc:374] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:54.692929 16168 raft_consensus.cc:729] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db46737b068146259087da995f1bb4fa, State: Initialized, Role: FOLLOWER
I20260928 16:48:54.693604 16168 consensus_queue.cc:260] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260928 16:48:54.693866 16168 raft_consensus.cc:388] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:54.694020 16168 raft_consensus.cc:482] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:54.694185 16168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:54.696458 16168 raft_consensus.cc:504] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260928 16:48:54.696986 16168 leader_election.cc:302] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db46737b068146259087da995f1bb4fa; no voters:
I20260928 16:48:54.697576 16168 leader_election.cc:288] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:54.697779 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:54.699371 16178 raft_consensus.cc:686] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [term 1 LEADER]: Becoming Leader. State: Replica: db46737b068146259087da995f1bb4fa, State: Running, Role: LEADER
I20260928 16:48:54.699924 16178 consensus_queue.cc:237] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } }
I20260928 16:48:54.700112 16168 sys_catalog.cc:564] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:54.702943 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db46737b068146259087da995f1bb4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } } }
I20260928 16:48:54.702867 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader db46737b068146259087da995f1bb4fa. Latest consensus state: current_term: 1 leader_uuid: "db46737b068146259087da995f1bb4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46737b068146259087da995f1bb4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34545 } } }
I20260928 16:48:54.703372 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:54.703428 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:54.705024 16186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:54.709748 16186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:54.718287 16186 catalog_manager.cc:1347] Generated new cluster ID: a4d9883d19aa4803b7bd36e62f91b80e
I20260928 16:48:54.718416 16186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:54.731197 16186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:54.731676 16186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:54.741143 16186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db46737b068146259087da995f1bb4fa: Generated new TSK 0
I20260928 16:48:54.741978 16186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:54.757588 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16101
W20260928 16:48:54.758927 16163 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:59545
W20260928 16:48:54.759878 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16101: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:54.760062 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16101
2026-09-28T16:48:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:54 dist-test-slave-t0tw krb5kdc[16199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:54 dist-test-slave-t0tw krb5kdc[16199](info): set up 2 sockets
Sep 28 16:48:54 dist-test-slave-t0tw krb5kdc[16199](info): commencing operation
krb5kdc: starting...
W20260928 16:48:55.787009 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:55 dist-test-slave-t0tw krb5kdc[16199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:56.055467 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:56.389896 16212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:56.390200 16212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:56.390323 16212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:56.397696 16212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:56.397817 16212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:56.397912 16212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:56.398003 16212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:56.398065 16212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:56.408160 16212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:56.409498 16212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:56.410858 16212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:56.422461 16218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:56.422431 16220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:56.422397 16217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:56.423619 16212 server_base.cc:1034] running on GCE node
I20260928 16:48:56.424495 16212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:56.425983 16212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:56.427196 16212 hybrid_clock.cc:648] HybridClock initialized: now 1790614136427145 us; error 38 us; skew 500 ppm
I20260928 16:48:56.427590 16212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:56 dist-test-slave-t0tw krb5kdc[16199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:56.430778 16212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:56.432273 16212 webserver.cc:460] Webserver started at http://127.0.0.1:40363/ using document root <none> and password file <none>
I20260928 16:48:56.432945 16212 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:56.433075 16212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:56.433370 16212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:56.436440 16212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "13f7e5222d8d42cbbad59e583e50693a"
format_stamp: "Formatted at 2026-09-28 16:48:56 on dist-test-slave-t0tw"
server_key: "d09d0f393e1000d92b1403ea08273fed"
server_key_iv: "847aca9a1c1245032102e6b8859d3dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:56.437218 16212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "13f7e5222d8d42cbbad59e583e50693a"
format_stamp: "Formatted at 2026-09-28 16:48:56 on dist-test-slave-t0tw"
server_key: "d09d0f393e1000d92b1403ea08273fed"
server_key_iv: "847aca9a1c1245032102e6b8859d3dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:56.442417 16212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:56.445963 16227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:56.447300 16212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:56.447474 16212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "13f7e5222d8d42cbbad59e583e50693a"
format_stamp: "Formatted at 2026-09-28 16:48:56 on dist-test-slave-t0tw"
server_key: "d09d0f393e1000d92b1403ea08273fed"
server_key_iv: "847aca9a1c1245032102e6b8859d3dc6"
server_key_version: "encryptionkey@0"
I20260928 16:48:56.447660 16212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:56.465675 16212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:56.470250 16212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:56.470539 16212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:56.487840 16212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33879
I20260928 16:48:56.487898 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33879 every 8 connection(s)
I20260928 16:48:56.489192 16212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:56.491888 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16212
I20260928 16:48:56.492064 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:56.492623 27028 external_mini_cluster.cc:1428] Setting key fab72513143a2af3013e29c0220d15c7
I20260928 16:48:56.493546 16279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 16:48:56 dist-test-slave-t0tw krb5kdc[16199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:48:56.508091 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: Bootstrap starting.
I20260928 16:48:56.511825 16279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:56.513075 16279 log.cc:824] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:56.516512 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: No bootstrap required, opened a new log
I20260928 16:48:56.517146 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:56.494490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:48293 (local address 127.0.0.1:33879)
0928 16:48:56.495293 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:56.495305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:56.495348 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:48:56.497000 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:56.497024 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:56.497429 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:56.497777 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:56.499688 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:56.501065 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:56.502787 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:56.503509 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:56.503661 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:56.508885 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:56.508894 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:56.508910 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:56.508955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:56.512341 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:56.513382 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:56.513389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:56.513396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:56.513580 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:56.514145 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:56.514150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:56.514152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:56.514821 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:56.515059 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:56.515757 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 16:48:56.516069 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":309,"threads_started":1}
I20260928 16:48:56.521230 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:56.526597 16279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } }
I20260928 16:48:56.526969 16279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:56.527148 16279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f7e5222d8d42cbbad59e583e50693a, State: Initialized, Role: FOLLOWER
I20260928 16:48:56.527709 16279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } }
I20260928 16:48:56.527937 16279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:56.528066 16279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:56.528226 16279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:56.530471 16279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } }
I20260928 16:48:56.531001 16279 leader_election.cc:302] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f7e5222d8d42cbbad59e583e50693a; no voters:
I20260928 16:48:56.531618 16279 leader_election.cc:288] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:56.531773 16289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:56.533334 16289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [term 1 LEADER]: Becoming Leader. State: Replica: 13f7e5222d8d42cbbad59e583e50693a, State: Running, Role: LEADER
I20260928 16:48:56.533825 16279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:56.533959 16289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } }
I20260928 16:48:56.536796 16290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f7e5222d8d42cbbad59e583e50693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } } }
I20260928 16:48:56.537169 16290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:56.536716 16291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f7e5222d8d42cbbad59e583e50693a. Latest consensus state: current_term: 1 leader_uuid: "13f7e5222d8d42cbbad59e583e50693a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f7e5222d8d42cbbad59e583e50693a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33879 } } }
I20260928 16:48:56.544795 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:56.526330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:37505 (local address 127.0.0.1:33879)
0928 16:48:56.526516 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:56.526522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:56.527033 (+ 511us) server_negotiation.cc:407] Connection header received
0928 16:48:56.527250 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:56.527256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:56.527337 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:56.527423 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:56.529169 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:56.529959 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:56.531356 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:56.532091 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:56.532235 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:56.535755 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:56.535764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:56.535769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:48:56.535813 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:56.539853 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:56.541039 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:56.541046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:56.541050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:56.541291 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:48:56.541908 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:48:56.541914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:48:56.541918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:48:56.542404 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:56.542730 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:56.544143 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0928 16:48:56.544356 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:48:56.545827 16291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a [sys.catalog]: This master's current role is: LEADER
W20260928 16:48:56.552074 16306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:48:56.552191 16306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:48:56.552336 16295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:56.554560 16295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:56.563549 16295 catalog_manager.cc:1347] Generated new cluster ID: 40abc633e6a84828ab52e99a1123a08f
I20260928 16:48:56.563723 16295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:56.584230 16295 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:56.586180 16295 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:56.606233 16295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f7e5222d8d42cbbad59e583e50693a: Generated new TSK 0
I20260928 16:48:56.607426 16295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:56.626433 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16212
I20260928 16:48:56.627677 16274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:48:56.722337 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16212
W20260928 16:48:56.738677 16197 connection.cc:535] client connection to 127.0.0.1:33879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33879): Connection reset by peer (error 104)
2026-09-28T16:48:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:56 dist-test-slave-t0tw krb5kdc[16312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:56 dist-test-slave-t0tw krb5kdc[16312](info): set up 2 sockets
Sep 28 16:48:56 dist-test-slave-t0tw krb5kdc[16312](info): commencing operation
krb5kdc: starting...
W20260928 16:48:57.751092 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:57 dist-test-slave-t0tw krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:58.024657 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:48:58.378189 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:48:58.378538 16325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:48:58.378685 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:48:58.386487 16325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:48:58.386612 16325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:48:58.386667 16325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:48:58.386750 16325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:48:58.386801 16325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:48:58.397133 16325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:48:58.398530 16325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:48:58.399977 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:48:58.412797 16330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:58.412765 16331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:48:58.412774 16333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:48:58.413528 16325 server_base.cc:1034] running on GCE node
I20260928 16:48:58.414386 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:48:58.416070 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:48:58.417341 16325 hybrid_clock.cc:648] HybridClock initialized: now 1790614138417283 us; error 47 us; skew 500 ppm
I20260928 16:48:58.417747 16325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:48:58 dist-test-slave-t0tw krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:48:58.421092 16325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:48:58.422971 16325 webserver.cc:460] Webserver started at http://127.0.0.1:37121/ using document root <none> and password file <none>
I20260928 16:48:58.423746 16325 fs_manager.cc:359] Metadata directory not provided
I20260928 16:48:58.423887 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:48:58.424209 16325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:48:58.427412 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "161386c0a9d54d9983f8cb85c776e034"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-t0tw"
server_key: "1a8507071d2f55cd85f091143e86d892"
server_key_iv: "10b7b2b6419b802a4bd3330bfc0df14d"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.428345 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "161386c0a9d54d9983f8cb85c776e034"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-t0tw"
server_key: "1a8507071d2f55cd85f091143e86d892"
server_key_iv: "10b7b2b6419b802a4bd3330bfc0df14d"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.433946 16325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:48:58.437705 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:48:58.439208 16325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:48:58.439379 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "161386c0a9d54d9983f8cb85c776e034"
format_stamp: "Formatted at 2026-09-28 16:48:58 on dist-test-slave-t0tw"
server_key: "1a8507071d2f55cd85f091143e86d892"
server_key_iv: "10b7b2b6419b802a4bd3330bfc0df14d"
server_key_version: "encryptionkey@0"
I20260928 16:48:58.439580 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:48:58.464228 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:48:58.468730 16325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:48:58.469022 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:48:58.487457 16325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260928 16:48:58.487506 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260928 16:48:58.488852 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:48:58.491456 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16325
I20260928 16:48:58.491613 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:48:58.492050 27028 external_mini_cluster.cc:1428] Setting key 30af2d2d37057fe7afdabb3e14acf2b8
I20260928 16:48:58.493027 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:48:58.506328 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:58.493998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:38805 (local address 127.0.0.1:43687)
0928 16:48:58.494782 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:58.494795 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:58.494836 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:48:58.496482 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:58.496508 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:58.496950 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:58.497309 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:58.498005 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.499418 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.501343 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.502116 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.502269 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:58.503723 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:58.503734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:58.503751 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:58.504442 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:58.504535 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:58.504566 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:58.504718 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:58.505088 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:48:58.505294 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":338,"threads_started":1}
I20260928 16:48:58.507171 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034: Bootstrap starting.
I20260928 16:48:58.510349 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:48:58.510489 16392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034: Neither blocks nor log segments found. Creating new log.
I20260928 16:48:58.511590 16392 log.cc:824] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034: Log is configured to *not* fsync() on all Append() calls
I20260928 16:48:58.514770 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034: No bootstrap required, opened a new log
I20260928 16:48:58.520951 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:48:58.514792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:58809 (local address 127.0.0.1:43687)
0928 16:48:58.515044 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:48:58.515050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:48:58.515374 (+ 324us) server_negotiation.cc:407] Connection header received
0928 16:48:58.515573 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:48:58.515577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:48:58.515639 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:48:58.515726 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:48:58.516231 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.517059 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.518396 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:48:58.518874 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:48:58.519020 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:48:58.520193 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:48:58.520198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:48:58.520201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:48:58.520219 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:48:58.520265 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:48:58.520284 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:48:58.520379 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:48:58.520602 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:48:58.520658 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:48:58.523209 16392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260928 16:48:58.523643 16392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:48:58.523748 16392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161386c0a9d54d9983f8cb85c776e034, State: Initialized, Role: FOLLOWER
I20260928 16:48:58.524365 16392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260928 16:48:58.524613 16392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:48:58.524751 16392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:48:58.524904 16392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:48:58.527168 16392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260928 16:48:58.527706 16392 leader_election.cc:302] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161386c0a9d54d9983f8cb85c776e034; no voters:
I20260928 16:48:58.528353 16392 leader_election.cc:288] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:48:58.528524 16402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:48:58.529753 16402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [term 1 LEADER]: Becoming Leader. State: Replica: 161386c0a9d54d9983f8cb85c776e034, State: Running, Role: LEADER
I20260928 16:48:58.530292 16402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260928 16:48:58.530758 16392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:48:58.533133 16403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161386c0a9d54d9983f8cb85c776e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260928 16:48:58.533198 16404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161386c0a9d54d9983f8cb85c776e034. Latest consensus state: current_term: 1 leader_uuid: "161386c0a9d54d9983f8cb85c776e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161386c0a9d54d9983f8cb85c776e034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260928 16:48:58.533496 16403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:58.533576 16404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034 [sys.catalog]: This master's current role is: LEADER
I20260928 16:48:58.534359 16411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:48:58.540508 16411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:48:58.553542 16411 catalog_manager.cc:1347] Generated new cluster ID: eebcf102ecbf423fa20acee6138dd777
I20260928 16:48:58.553682 16411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:48:58.569931 16411 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:48:58.571841 16411 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:48:58.581319 16411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161386c0a9d54d9983f8cb85c776e034: Generated new TSK 0
I20260928 16:48:58.582230 16411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:48:58.599473 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16325
W20260928 16:48:58.600750 16387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:38805
W20260928 16:48:58.601497 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:48:58.601671 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16325
W20260928 16:48:58.617231 16310 connection.cc:535] client connection to 127.0.0.1:43687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43687): Connection reset by peer (error 104)
2026-09-28T16:48:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:48:58 dist-test-slave-t0tw krb5kdc[16423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:48:58 dist-test-slave-t0tw krb5kdc[16423](info): set up 2 sockets
Sep 28 16:48:58 dist-test-slave-t0tw krb5kdc[16423](info): commencing operation
krb5kdc: starting...
W20260928 16:48:59.639329 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:48:59 dist-test-slave-t0tw krb5kdc[16423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:48:59.911392 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:00.248929 16436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:00.249279 16436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:00.249428 16436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:00.256844 16436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:00.256994 16436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:00.257084 16436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:00.257175 16436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:00.257236 16436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:00.267500 16436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:00.268857 16436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:00.270185 16436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:00.283131 16441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:00.283188 16444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:00.283249 16442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:00.284301 16436 server_base.cc:1034] running on GCE node
I20260928 16:49:00.285162 16436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:00.286680 16436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:00.287884 16436 hybrid_clock.cc:648] HybridClock initialized: now 1790614140287819 us; error 58 us; skew 500 ppm
I20260928 16:49:00.288239 16436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:00 dist-test-slave-t0tw krb5kdc[16423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:00.298324 16436 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:49:00.300208 16436 webserver.cc:460] Webserver started at http://127.0.0.1:34289/ using document root <none> and password file <none>
I20260928 16:49:00.301003 16436 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:00.301134 16436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:00.301460 16436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:00.304751 16436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "b689946c1bb24f01ad7a3778f1418091"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-t0tw"
server_key: "ef9784d3875e520859e84fd2ec69196a"
server_key_iv: "e4e710c756633a5fa5c95855b37f580c"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.305576 16436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "b689946c1bb24f01ad7a3778f1418091"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-t0tw"
server_key: "ef9784d3875e520859e84fd2ec69196a"
server_key_iv: "e4e710c756633a5fa5c95855b37f580c"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.311229 16436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:49:00.314841 16451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:00.316326 16436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:00.316514 16436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "b689946c1bb24f01ad7a3778f1418091"
format_stamp: "Formatted at 2026-09-28 16:49:00 on dist-test-slave-t0tw"
server_key: "ef9784d3875e520859e84fd2ec69196a"
server_key_iv: "e4e710c756633a5fa5c95855b37f580c"
server_key_version: "encryptionkey@0"
I20260928 16:49:00.316717 16436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:00.322512 16436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:00.327013 16436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:00.327360 16436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:00.345242 16436 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36573
I20260928 16:49:00.345270 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36573 every 8 connection(s)
I20260928 16:49:00.346715 16436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:00.346980 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16436
I20260928 16:49:00.347178 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:00.347659 27028 external_mini_cluster.cc:1428] Setting key c5bdaef9ad74782273c265f8c6433340
I20260928 16:49:00.351145 16503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:00.355682 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:00.349608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:52419 (local address 127.0.0.1:36573)
0928 16:49:00.350374 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:00.350386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:00.350429 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:49:00.352140 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:00.352163 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:00.352494 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:00.352814 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:00.353927 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:00.353935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:00.353948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:00.353968 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:00.354066 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:00.354092 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:00.354284 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:00.354410 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:49:00.354606 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
I20260928 16:49:00.359736 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:00.363112 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091: Bootstrap starting.
I20260928 16:49:00.366470 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:00.363979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:34743 (local address 127.0.0.1:36573)
0928 16:49:00.364137 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:00.364142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:00.364636 (+ 494us) server_negotiation.cc:407] Connection header received
0928 16:49:00.364809 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:00.364815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:00.364876 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:00.364975 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:00.365839 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:00.365845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:00.365850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:00.365867 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:00.365910 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:00.365926 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:00.366029 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:00.366134 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:49:00.366196 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:49:00.366465 16503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:00.367780 16503 log.cc:824] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:00.370589 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091: No bootstrap required, opened a new log
I20260928 16:49:00.378060 16503 raft_consensus.cc:348] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260928 16:49:00.378453 16503 raft_consensus.cc:374] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:00.378554 16503 raft_consensus.cc:729] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b689946c1bb24f01ad7a3778f1418091, State: Initialized, Role: FOLLOWER
I20260928 16:49:00.379230 16503 consensus_queue.cc:260] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260928 16:49:00.379480 16503 raft_consensus.cc:388] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:00.379616 16503 raft_consensus.cc:482] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:00.379782 16503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:00.381978 16503 raft_consensus.cc:504] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260928 16:49:00.382499 16503 leader_election.cc:302] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b689946c1bb24f01ad7a3778f1418091; no voters:
I20260928 16:49:00.383149 16503 leader_election.cc:288] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:00.383294 16513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:00.383965 16513 raft_consensus.cc:686] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [term 1 LEADER]: Becoming Leader. State: Replica: b689946c1bb24f01ad7a3778f1418091, State: Running, Role: LEADER
I20260928 16:49:00.384505 16513 consensus_queue.cc:237] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260928 16:49:00.385331 16503 sys_catalog.cc:564] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:00.387372 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b689946c1bb24f01ad7a3778f1418091. Latest consensus state: current_term: 1 leader_uuid: "b689946c1bb24f01ad7a3778f1418091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } } }
I20260928 16:49:00.387713 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:00.389303 16519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:00.389731 16514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b689946c1bb24f01ad7a3778f1418091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b689946c1bb24f01ad7a3778f1418091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } } }
I20260928 16:49:00.390089 16514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:00.394721 16519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:00.404513 16519 catalog_manager.cc:1347] Generated new cluster ID: d34c4821e0394b37be07eafe28cd1344
I20260928 16:49:00.404625 16519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:00.446084 16519 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:00.446540 16519 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:00.456375 16519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b689946c1bb24f01ad7a3778f1418091: Generated new TSK 0
I20260928 16:49:00.457203 16519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:00.468420 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16436
W20260928 16:49:00.469928 16498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:52419
W20260928 16:49:00.470621 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16436: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:00.470753 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16436
2026-09-28T16:49:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:00 dist-test-slave-t0tw krb5kdc[16534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:00 dist-test-slave-t0tw krb5kdc[16534](info): set up 2 sockets
Sep 28 16:49:00 dist-test-slave-t0tw krb5kdc[16534](info): commencing operation
krb5kdc: starting...
W20260928 16:49:01.507045 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:01 dist-test-slave-t0tw krb5kdc[16534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:01.770804 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41079
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:02.109380 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:02.109694 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:02.109813 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:02.117281 16547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:02.117425 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:02.117476 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:02.117578 16547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:02.117640 16547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:02.128002 16547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:41079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:02.129366 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:02.130733 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:02.143414 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:02.143376 16555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:02.143334 16553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:02.144802 16547 server_base.cc:1034] running on GCE node
I20260928 16:49:02.145780 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:02.147360 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:02.148617 16547 hybrid_clock.cc:648] HybridClock initialized: now 1790614142148564 us; error 48 us; skew 500 ppm
I20260928 16:49:02.149027 16547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:02 dist-test-slave-t0tw krb5kdc[16534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614142, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:02.152701 16547 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:02.154479 16547 webserver.cc:460] Webserver started at http://127.26.101.62:34565/ using document root <none> and password file <none>
I20260928 16:49:02.155359 16547 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:02.155580 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:02.155915 16547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:02.160346 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba"
format_stamp: "Formatted at 2026-09-28 16:49:02 on dist-test-slave-t0tw"
server_key: "f7ebf54dd06620322e0274d3f1a20e18"
server_key_iv: "391bb2c6d260931fa763b8e7b8bea5ac"
server_key_version: "encryptionkey@0"
I20260928 16:49:02.161463 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba"
format_stamp: "Formatted at 2026-09-28 16:49:02 on dist-test-slave-t0tw"
server_key: "f7ebf54dd06620322e0274d3f1a20e18"
server_key_iv: "391bb2c6d260931fa763b8e7b8bea5ac"
server_key_version: "encryptionkey@0"
I20260928 16:49:02.167933 16547 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 16:49:02.172042 16562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:02.173523 16547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:49:02.173722 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba"
format_stamp: "Formatted at 2026-09-28 16:49:02 on dist-test-slave-t0tw"
server_key: "f7ebf54dd06620322e0274d3f1a20e18"
server_key_iv: "391bb2c6d260931fa763b8e7b8bea5ac"
server_key_version: "encryptionkey@0"
I20260928 16:49:02.173930 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:02.191792 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:02.196552 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:02.196842 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:02.215330 16547 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:41079
I20260928 16:49:02.215363 16613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:41079 every 8 connection(s)
I20260928 16:49:02.216804 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:02.220744 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:02.227238 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16547
I20260928 16:49:02.227423 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:02.227861 27028 external_mini_cluster.cc:1428] Setting key ddc1df67fa4c0a1804285ef9db882432
I20260928 16:49:02.232290 16614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba: Bootstrap starting.
I20260928 16:49:02.236135 16614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:02.237340 16614 log.cc:824] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:02.240368 16614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba: No bootstrap required, opened a new log
Sep 28 16:49:02 dist-test-slave-t0tw krb5kdc[16534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:02.248278 16614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } }
I20260928 16:49:02.248677 16614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:02.248773 16614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427a7a1fa7b4d6182bc9f4891f8d1ba, State: Initialized, Role: FOLLOWER
I20260928 16:49:02.249430 16614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } }
I20260928 16:49:02.249681 16614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:02.249814 16614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:02.249986 16614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:02.251624 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:02.229941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43731 (local address 127.26.101.62:41079)
0928 16:49:02.230802 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:02.230815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:02.230866 (+ 51us) server_negotiation.cc:407] Connection header received
0928 16:49:02.232693 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:02.232721 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:02.233157 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:02.233512 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:02.235465 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:02.236866 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:02.238570 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:02.239324 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:02.239502 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:02.243692 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:02.243703 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:02.243723 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:02.243767 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:02.246990 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:02.247892 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:02.247899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:02.247906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:02.248117 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:02.248592 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:02.248596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:02.248598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:02.249246 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:02.249586 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:02.250109 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 16:49:02.250445 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":396,"threads_started":1}
I20260928 16:49:02.252627 16614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } }
I20260928 16:49:02.253168 16614 leader_election.cc:302] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2427a7a1fa7b4d6182bc9f4891f8d1ba; no voters:
I20260928 16:49:02.253780 16614 leader_election.cc:288] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:02.253957 16619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:02.255043 16619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [term 1 LEADER]: Becoming Leader. State: Replica: 2427a7a1fa7b4d6182bc9f4891f8d1ba, State: Running, Role: LEADER
I20260928 16:49:02.255688 16619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } }
I20260928 16:49:02.255980 16614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:02.256358 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:02.262473 16621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2427a7a1fa7b4d6182bc9f4891f8d1ba. Latest consensus state: current_term: 1 leader_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } } }
I20260928 16:49:02.262981 16621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:02.262461 16620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427a7a1fa7b4d6182bc9f4891f8d1ba" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41079 } } }
I20260928 16:49:02.263964 16620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:02.271667 16629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:02.283210 16629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:02.291272 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:02.273996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33979 (local address 127.26.101.62:41079)
0928 16:49:02.274356 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:02.274363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:02.274697 (+ 334us) server_negotiation.cc:407] Connection header received
0928 16:49:02.274884 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:02.274890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:02.274963 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:02.275059 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:02.276533 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:02.277365 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:02.278889 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:02.279543 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:02.279676 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:02.283228 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:02.283236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:02.283241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:02.283285 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:02.287598 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:02.288758 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:02.288767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:02.288771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:02.288918 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:02.289501 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:02.289507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:02.289511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:02.289949 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:02.290221 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:02.290703 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 16:49:02.290897 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260928 16:49:02.302104 16629 catalog_manager.cc:1347] Generated new cluster ID: 51df0e6164014779aea154dcc944a929
I20260928 16:49:02.302273 16629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:02.323180 16629 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:02.324563 16629 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:02.339393 16629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2427a7a1fa7b4d6182bc9f4891f8d1ba: Generated new TSK 0
I20260928 16:49:02.340284 16629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:02.346765 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16547
I20260928 16:49:02.348567 16609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:02.443934 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16547
W20260928 16:49:02.460108 16532 connection.cc:535] client connection to 127.26.101.62:41079 recv error: Network error: failed to read from TLS socket (remote: 127.26.101.62:41079): Connection reset by peer (error 104)
2026-09-28T16:49:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:02 dist-test-slave-t0tw krb5kdc[16646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:02 dist-test-slave-t0tw krb5kdc[16646](info): set up 2 sockets
Sep 28 16:49:02 dist-test-slave-t0tw krb5kdc[16646](info): commencing operation
krb5kdc: starting...
W20260928 16:49:03.487126 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:03 dist-test-slave-t0tw krb5kdc[16646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:03.755167 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45379
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:04.100379 16659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:04.100696 16659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:04.100816 16659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:04.108263 16659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:04.108410 16659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:04.108459 16659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:04.108544 16659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:04.108608 16659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:04.118830 16659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:45379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:04.120252 16659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:04.121603 16659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:04.133527 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:04.133438 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:04.133502 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:04.134581 16659 server_base.cc:1034] running on GCE node
I20260928 16:49:04.135509 16659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:04.137030 16659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:04.138240 16659 hybrid_clock.cc:648] HybridClock initialized: now 1790614144138186 us; error 44 us; skew 500 ppm
I20260928 16:49:04.138603 16659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:04 dist-test-slave-t0tw krb5kdc[16646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614144, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:04.141870 16659 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:04.143512 16659 webserver.cc:460] Webserver started at http://127.26.101.62:44145/ using document root <none> and password file <none>
I20260928 16:49:04.144235 16659 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:04.144356 16659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:04.144650 16659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:04.147984 16659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "927b7a9d13dd45888fc7d3145d797ad1"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-t0tw"
server_key: "5abe910f04a605cb5e2b7ce4d300bdad"
server_key_iv: "844c0c7d1984340d4bdf815ad442556d"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.148818 16659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "927b7a9d13dd45888fc7d3145d797ad1"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-t0tw"
server_key: "5abe910f04a605cb5e2b7ce4d300bdad"
server_key_iv: "844c0c7d1984340d4bdf815ad442556d"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.154300 16659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:49:04.158602 16674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:04.160411 16659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:49:04.160687 16659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "927b7a9d13dd45888fc7d3145d797ad1"
format_stamp: "Formatted at 2026-09-28 16:49:04 on dist-test-slave-t0tw"
server_key: "5abe910f04a605cb5e2b7ce4d300bdad"
server_key_iv: "844c0c7d1984340d4bdf815ad442556d"
server_key_version: "encryptionkey@0"
I20260928 16:49:04.160955 16659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:04.187345 16659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:04.192149 16659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:04.192466 16659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:04.210731 16659 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:45379
I20260928 16:49:04.210754 16725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:45379 every 8 connection(s)
I20260928 16:49:04.212267 16659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:04.216157 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:04.222249 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16659
I20260928 16:49:04.222424 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:04.222853 27028 external_mini_cluster.cc:1428] Setting key 7094bb252e8c2fe1740156cef92a9787
I20260928 16:49:04.227468 16726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1: Bootstrap starting.
I20260928 16:49:04.230695 16726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:04.231827 16726 log.cc:824] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:04.235391 16726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1: No bootstrap required, opened a new log
I20260928 16:49:04.236616 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:04.224775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44535 (local address 127.26.101.62:45379)
0928 16:49:04.225630 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:04.225643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:04.225685 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:49:04.227398 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:04.227423 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:04.227874 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:04.228252 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:04.228795 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:04.230161 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:04.231630 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:04.232428 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:04.232678 (+ 250us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:04.233808 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:04.233819 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:04.233839 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:04.234569 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:04.234674 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:04.234708 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:04.234877 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:04.235125 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 16:49:04.235315 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":364,"threads_started":1}
I20260928 16:49:04.240494 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:04.243561 16726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } }
I20260928 16:49:04.244077 16726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:04.244196 16726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927b7a9d13dd45888fc7d3145d797ad1, State: Initialized, Role: FOLLOWER
I20260928 16:49:04.244899 16726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } }
I20260928 16:49:04.245203 16726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:04.245352 16726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:04.245550 16726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:04.248133 16726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } }
I20260928 16:49:04.248703 16726 leader_election.cc:302] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927b7a9d13dd45888fc7d3145d797ad1; no voters:
I20260928 16:49:04.249421 16726 leader_election.cc:288] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:04.249584 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:04.250994 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 927b7a9d13dd45888fc7d3145d797ad1, State: Running, Role: LEADER
I20260928 16:49:04.251672 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } }
I20260928 16:49:04.251981 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:04.244884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34815 (local address 127.26.101.62:45379)
0928 16:49:04.245084 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:04.245094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:04.245533 (+ 439us) server_negotiation.cc:407] Connection header received
0928 16:49:04.245745 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:04.245750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:04.245814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:04.245904 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:04.246412 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:04.247247 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:04.248700 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:04.249222 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:04.249340 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:04.250840 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:04.250846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:04.250851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:04.250879 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:04.250938 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:04.250960 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:04.251135 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:04.251532 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 16:49:04.251596 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:49:04.252095 16726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:04.254889 16737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "927b7a9d13dd45888fc7d3145d797ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } } }
I20260928 16:49:04.255585 16737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:04.254914 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927b7a9d13dd45888fc7d3145d797ad1. Latest consensus state: current_term: 1 leader_uuid: "927b7a9d13dd45888fc7d3145d797ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927b7a9d13dd45888fc7d3145d797ad1" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 45379 } } }
I20260928 16:49:04.256211 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:04.257334 16745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:04.266819 16745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:04.276824 16745 catalog_manager.cc:1347] Generated new cluster ID: bd6daba8e51d4ebdad010c677abb36a5
I20260928 16:49:04.276983 16745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:04.291985 16745 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:04.293812 16745 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:04.305670 16745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 927b7a9d13dd45888fc7d3145d797ad1: Generated new TSK 0
I20260928 16:49:04.306577 16745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:04.311743 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16659
W20260928 16:49:04.312956 16721 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44535
W20260928 16:49:04.313791 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:04.313942 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16659
W20260928 16:49:04.329339 16644 connection.cc:535] client connection to 127.26.101.62:45379 recv error: Network error: failed to read from TLS socket (remote: 127.26.101.62:45379): Connection reset by peer (error 104)
2026-09-28T16:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:04 dist-test-slave-t0tw krb5kdc[16757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:04 dist-test-slave-t0tw krb5kdc[16757](info): set up 2 sockets
Sep 28 16:49:04 dist-test-slave-t0tw krb5kdc[16757](info): commencing operation
krb5kdc: starting...
W20260928 16:49:05.343109 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:05 dist-test-slave-t0tw krb5kdc[16757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:05.614681 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44155
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:44155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:05.954102 16770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:05.954440 16770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:05.954584 16770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:05.961982 16770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:05.962138 16770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:05.962224 16770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:05.962306 16770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:05.962363 16770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:05.972723 16770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:44155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:05.974133 16770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:05.975580 16770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:05.988119 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:05.988166 16778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:05.988190 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:05.989032 16770 server_base.cc:1034] running on GCE node
I20260928 16:49:05.989907 16770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:05.991521 16770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:05.992774 16770 hybrid_clock.cc:648] HybridClock initialized: now 1790614145992703 us; error 84 us; skew 500 ppm
I20260928 16:49:05.993162 16770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:05 dist-test-slave-t0tw krb5kdc[16757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614145, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:06.001320 16770 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:06.003021 16770 webserver.cc:460] Webserver started at http://127.26.101.62:33883/ using document root <none> and password file <none>
I20260928 16:49:06.003789 16770 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:06.003917 16770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:06.004225 16770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:06.007591 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "0ee57ab45a5949209b0f965f2f9f1081"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-t0tw"
server_key: "0d577aed483c3d1d8352f0727544cbec"
server_key_iv: "713e0deddc68e4f2651ba68146dc5635"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.008422 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "0ee57ab45a5949209b0f965f2f9f1081"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-t0tw"
server_key: "0d577aed483c3d1d8352f0727544cbec"
server_key_iv: "713e0deddc68e4f2651ba68146dc5635"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.014067 16770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 16:49:06.017992 16785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:06.019588 16770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:06.019779 16770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "0ee57ab45a5949209b0f965f2f9f1081"
format_stamp: "Formatted at 2026-09-28 16:49:06 on dist-test-slave-t0tw"
server_key: "0d577aed483c3d1d8352f0727544cbec"
server_key_iv: "713e0deddc68e4f2651ba68146dc5635"
server_key_version: "encryptionkey@0"
I20260928 16:49:06.020042 16770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:06.026549 16770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:06.031147 16770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:06.031452 16770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:06.050051 16770 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:44155
I20260928 16:49:06.050086 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:44155 every 8 connection(s)
I20260928 16:49:06.051640 16770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:06.055543 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:06.060901 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16770
I20260928 16:49:06.061081 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:06.061533 27028 external_mini_cluster.cc:1428] Setting key 277d50c762161737a978da585f6ee1c6
I20260928 16:49:06.068572 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081: Bootstrap starting.
I20260928 16:49:06.071403 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.063574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58243 (local address 127.26.101.62:44155)
0928 16:49:06.064490 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.064509 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.064574 (+ 65us) server_negotiation.cc:407] Connection header received
0928 16:49:06.067110 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.067150 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.067734 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.068254 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.069287 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.069295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.069313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:06.069330 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.069420 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:06.069450 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.069667 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.069797 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.070019 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":397,"threads_started":1}
I20260928 16:49:06.072602 16837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:06.073719 16837 log.cc:824] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:06.075446 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:06.076797 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081: No bootstrap required, opened a new log
I20260928 16:49:06.082307 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:06.079812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56755 (local address 127.26.101.62:44155)
0928 16:49:06.080098 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:06.080104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:06.080368 (+ 264us) server_negotiation.cc:407] Connection header received
0928 16:49:06.080550 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:06.080559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:06.080610 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:06.080694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:06.081628 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:06.081633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:06.081637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:06.081655 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:06.081700 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:06.081716 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:06.081824 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:06.081957 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:49:06.082035 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 16:49:06.085433 16837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } }
I20260928 16:49:06.085851 16837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:06.085950 16837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee57ab45a5949209b0f965f2f9f1081, State: Initialized, Role: FOLLOWER
I20260928 16:49:06.086525 16837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } }
I20260928 16:49:06.086778 16837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:06.086923 16837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:06.087112 16837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:06.089490 16837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } }
I20260928 16:49:06.090145 16837 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ee57ab45a5949209b0f965f2f9f1081; no voters:
I20260928 16:49:06.090819 16837 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:06.090950 16847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:06.091619 16847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee57ab45a5949209b0f965f2f9f1081, State: Running, Role: LEADER
I20260928 16:49:06.092204 16847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } }
I20260928 16:49:06.092903 16837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:06.095198 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ee57ab45a5949209b0f965f2f9f1081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } } }
I20260928 16:49:06.095645 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:06.098028 16855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:06.097963 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee57ab45a5949209b0f965f2f9f1081. Latest consensus state: current_term: 1 leader_uuid: "0ee57ab45a5949209b0f965f2f9f1081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee57ab45a5949209b0f965f2f9f1081" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 44155 } } }
I20260928 16:49:06.098425 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:06.103282 16855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:06.112056 16855 catalog_manager.cc:1347] Generated new cluster ID: 84397499ccbb47259b16c9475c51b90c
I20260928 16:49:06.112185 16855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:06.131052 16855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:06.131592 16855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:06.143169 16855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ee57ab45a5949209b0f965f2f9f1081: Generated new TSK 0
I20260928 16:49:06.144291 16855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:06.149515 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16770
W20260928 16:49:06.150744 16832 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58243
W20260928 16:49:06.151525 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16770: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:06.151705 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16770
2026-09-28T16:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:06 dist-test-slave-t0tw krb5kdc[16868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:06 dist-test-slave-t0tw krb5kdc[16868](info): set up 2 sockets
Sep 28 16:49:06 dist-test-slave-t0tw krb5kdc[16868](info): commencing operation
krb5kdc: starting...
W20260928 16:49:07.191201 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:07 dist-test-slave-t0tw krb5kdc[16868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:07.455691 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41365
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:07.795650 16881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:07.795976 16881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:07.796114 16881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:07.803745 16881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:07.803872 16881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:07.803928 16881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:07.804013 16881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:07.804081 16881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:07.814627 16881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:41365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:07.816016 16881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:07.817354 16881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:07.829521 16886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:07.829532 16887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:07.829741 16889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:07.830263 16881 server_base.cc:1034] running on GCE node
I20260928 16:49:07.831260 16881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:07.832830 16881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:07.834066 16881 hybrid_clock.cc:648] HybridClock initialized: now 1790614147834021 us; error 40 us; skew 500 ppm
I20260928 16:49:07.834437 16881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:07 dist-test-slave-t0tw krb5kdc[16868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614147, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:07.837703 16881 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:07.839443 16881 webserver.cc:460] Webserver started at http://127.26.101.62:34283/ using document root <none> and password file <none>
I20260928 16:49:07.840170 16881 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:07.840296 16881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:07.840595 16881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:07.843854 16881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "8600535f783c48bf9af9f7394634f1a4"
format_stamp: "Formatted at 2026-09-28 16:49:07 on dist-test-slave-t0tw"
server_key: "3ff1e24aa1adff394e493233ea73caf1"
server_key_iv: "7c7cf4a24b42d95f90f196b74f0ce5ee"
server_key_version: "encryptionkey@0"
I20260928 16:49:07.844643 16881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "8600535f783c48bf9af9f7394634f1a4"
format_stamp: "Formatted at 2026-09-28 16:49:07 on dist-test-slave-t0tw"
server_key: "3ff1e24aa1adff394e493233ea73caf1"
server_key_iv: "7c7cf4a24b42d95f90f196b74f0ce5ee"
server_key_version: "encryptionkey@0"
I20260928 16:49:07.849993 16881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:49:07.853525 16896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:07.854828 16881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:49:07.855011 16881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "8600535f783c48bf9af9f7394634f1a4"
format_stamp: "Formatted at 2026-09-28 16:49:07 on dist-test-slave-t0tw"
server_key: "3ff1e24aa1adff394e493233ea73caf1"
server_key_iv: "7c7cf4a24b42d95f90f196b74f0ce5ee"
server_key_version: "encryptionkey@0"
I20260928 16:49:07.855243 16881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:07.875620 16881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:07.879787 16881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:07.880077 16881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:07.897990 16881 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:41365
I20260928 16:49:07.898010 16947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:41365 every 8 connection(s)
I20260928 16:49:07.899410 16881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:07.902391 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16881
I20260928 16:49:07.902554 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:07.902997 27028 external_mini_cluster.cc:1428] Setting key 15dbc8608b87d51364631819c059e0db
I20260928 16:49:07.903452 16948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:07.917397 16948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4: Bootstrap starting.
Sep 28 16:49:07 dist-test-slave-t0tw krb5kdc[16868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:07.921276 16948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:07.922681 16948 log.cc:824] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:07.926404 16948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4: No bootstrap required, opened a new log
I20260928 16:49:07.928262 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.904788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35979 (local address 127.26.101.62:41365)
0928 16:49:07.905589 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.905603 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.905653 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:49:07.908109 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.908140 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.908670 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.909136 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:07.910416 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.912392 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.913668 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.914752 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.914957 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.919148 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:07.919161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:07.919184 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:07.919245 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:07.923168 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:07.924232 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:07.924241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:07.924250 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:07.924466 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:07.924903 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:07.924907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:07.924910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:07.925641 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:07.925953 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.926332 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.926804 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":314,"threads_started":1}
I20260928 16:49:07.932495 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:07.935302 16948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } }
I20260928 16:49:07.935724 16948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:07.935828 16948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8600535f783c48bf9af9f7394634f1a4, State: Initialized, Role: FOLLOWER
I20260928 16:49:07.936393 16948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } }
I20260928 16:49:07.936695 16948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:07.936913 16948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:07.937137 16948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:07.939908 16948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } }
I20260928 16:49:07.940577 16948 leader_election.cc:302] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8600535f783c48bf9af9f7394634f1a4; no voters:
I20260928 16:49:07.941267 16948 leader_election.cc:288] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:07.941373 16958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:07.942533 16958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8600535f783c48bf9af9f7394634f1a4, State: Running, Role: LEADER
I20260928 16:49:07.943253 16958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } }
I20260928 16:49:07.943713 16948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:07.946524 16959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8600535f783c48bf9af9f7394634f1a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } } }
I20260928 16:49:07.946930 16959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:07.951752 16960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8600535f783c48bf9af9f7394634f1a4. Latest consensus state: current_term: 1 leader_uuid: "8600535f783c48bf9af9f7394634f1a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8600535f783c48bf9af9f7394634f1a4" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 41365 } } }
I20260928 16:49:07.952085 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:07.937004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51707 (local address 127.26.101.62:41365)
0928 16:49:07.937186 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:07.937193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:07.937726 (+ 533us) server_negotiation.cc:407] Connection header received
0928 16:49:07.937927 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:07.937932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:07.938023 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:07.938137 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:07.939724 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.940742 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.942272 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:07.942917 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:07.942996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:07.945933 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:07.945940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:07.945946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:07.945988 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:07.949197 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:07.949996 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:07.950002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:07.950005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:07.950129 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:07.950503 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:07.950506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:07.950508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:07.950853 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:07.951142 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:07.951466 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 16:49:07.951682 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:49:07.952139 16960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:07.955606 16965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:07.961330 16965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:07.971115 16965 catalog_manager.cc:1347] Generated new cluster ID: 2fa9ec6ca489434eb8797b4eff1a85ea
I20260928 16:49:07.971242 16965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:08.006400 16965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:08.008039 16965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:08.022028 16965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8600535f783c48bf9af9f7394634f1a4: Generated new TSK 0
I20260928 16:49:08.023016 16965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:08.034850 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16881
I20260928 16:49:08.036144 16943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:08.133028 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16881
2026-09-28T16:49:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:08 dist-test-slave-t0tw krb5kdc[16980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:08 dist-test-slave-t0tw krb5kdc[16980](info): set up 2 sockets
Sep 28 16:49:08 dist-test-slave-t0tw krb5kdc[16980](info): commencing operation
krb5kdc: starting...
W20260928 16:49:09.167308 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:09 dist-test-slave-t0tw krb5kdc[16980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:09.434501 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46713
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:09.772737 16993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:09.773082 16993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:09.773228 16993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:09.780805 16993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:09.780959 16993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:09.781047 16993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:09.781131 16993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:09.781193 16993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:09.791951 16993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:09.793344 16993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:09.794687 16993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:09.806612 16998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.806559 17001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:09.806589 16999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:09.807900 16993 server_base.cc:1034] running on GCE node
I20260928 16:49:09.808779 16993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:09.810344 16993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:09.811580 16993 hybrid_clock.cc:648] HybridClock initialized: now 1790614149811518 us; error 43 us; skew 500 ppm
I20260928 16:49:09.811986 16993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:09 dist-test-slave-t0tw krb5kdc[16980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614149, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:09.815395 16993 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:09.817011 16993 webserver.cc:460] Webserver started at http://127.26.101.62:39121/ using document root <none> and password file <none>
I20260928 16:49:09.817919 16993 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:09.818073 16993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:09.818462 16993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:09.821659 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "45fa42ccf2d14319af9259914ac43cd2"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-t0tw"
server_key: "37161a1d060a828c8be0c8d6700ff695"
server_key_iv: "4cc91a3773d4b8ae910c4a2cac447375"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.822431 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "45fa42ccf2d14319af9259914ac43cd2"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-t0tw"
server_key: "37161a1d060a828c8be0c8d6700ff695"
server_key_iv: "4cc91a3773d4b8ae910c4a2cac447375"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.828047 16993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:49:09.832039 17008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:09.833513 16993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:49:09.833683 16993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "45fa42ccf2d14319af9259914ac43cd2"
format_stamp: "Formatted at 2026-09-28 16:49:09 on dist-test-slave-t0tw"
server_key: "37161a1d060a828c8be0c8d6700ff695"
server_key_iv: "4cc91a3773d4b8ae910c4a2cac447375"
server_key_version: "encryptionkey@0"
I20260928 16:49:09.833863 16993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:09.855584 16993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:09.860165 16993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:09.860428 16993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:09.879923 16993 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:46713
I20260928 16:49:09.879961 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:46713 every 8 connection(s)
I20260928 16:49:09.881417 16993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:09.885501 17060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:09.890699 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 16993
I20260928 16:49:09.890875 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:09.891395 27028 external_mini_cluster.cc:1428] Setting key 1d3c30372c20a8a6a1cae2fc5a25dcbf
I20260928 16:49:09.897786 17060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: Bootstrap starting.
I20260928 16:49:09.901855 17060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:09.903383 17060 log.cc:824] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:09.906445 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.893381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54569 (local address 127.26.101.62:46713)
0928 16:49:09.894202 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.894214 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.894263 (+ 49us) server_negotiation.cc:407] Connection header received
0928 16:49:09.896171 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.896202 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.896673 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.897033 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.897703 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.899155 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.901211 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.901997 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.902200 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.903654 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.903666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.903685 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:09.904501 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.904617 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:09.904661 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.904804 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.905053 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.905360 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":370,"threads_started":1}
I20260928 16:49:09.906975 17060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: No bootstrap required, opened a new log
I20260928 16:49:09.910674 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:09.917048 17060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } }
I20260928 16:49:09.917560 17060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:09.917670 17060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fa42ccf2d14319af9259914ac43cd2, State: Initialized, Role: FOLLOWER
I20260928 16:49:09.918448 17060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } }
I20260928 16:49:09.918756 17060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:09.918926 17060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:09.919143 17060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:09.921857 17060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } }
I20260928 16:49:09.922484 17060 leader_election.cc:302] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fa42ccf2d14319af9259914ac43cd2; no voters:
I20260928 16:49:09.922894 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:09.916266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58139 (local address 127.26.101.62:46713)
0928 16:49:09.916467 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:09.916474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:09.916827 (+ 353us) server_negotiation.cc:407] Connection header received
0928 16:49:09.916986 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:09.916991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:09.917086 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:09.917209 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:09.917742 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.918809 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.920263 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:09.920943 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:09.921077 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:09.922156 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:09.922162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:09.922167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:09.922189 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:09.922243 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:09.922265 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:09.922353 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:09.922487 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 16:49:09.922589 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:49:09.923413 17060 leader_election.cc:288] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:09.923540 17070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:09.925933 17060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:09.926486 17070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 45fa42ccf2d14319af9259914ac43cd2, State: Running, Role: LEADER
I20260928 16:49:09.927132 17070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } }
I20260928 16:49:09.930341 17072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fa42ccf2d14319af9259914ac43cd2. Latest consensus state: current_term: 1 leader_uuid: "45fa42ccf2d14319af9259914ac43cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } } }
I20260928 16:49:09.930801 17072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:09.932029 17071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fa42ccf2d14319af9259914ac43cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fa42ccf2d14319af9259914ac43cd2" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 46713 } } }
I20260928 16:49:09.932328 17071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2 [sys.catalog]: This master's current role is: LEADER
W20260928 16:49:09.939879 17084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:09.940023 17084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:09.940153 17085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:09.942395 17085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:09.952200 17085 catalog_manager.cc:1347] Generated new cluster ID: 39ef8acb3e384ec89673a85fb34bd798
I20260928 16:49:09.952338 17085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:09.965524 17085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:09.966912 17085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:09.974138 17085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fa42ccf2d14319af9259914ac43cd2: Generated new TSK 0
I20260928 16:49:09.975144 17085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:09.990273 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16993
W20260928 16:49:09.991513 17055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54569
W20260928 16:49:09.992216 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 16993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:09.992380 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 16993
2026-09-28T16:49:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:10 dist-test-slave-t0tw krb5kdc[17091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:10 dist-test-slave-t0tw krb5kdc[17091](info): set up 2 sockets
Sep 28 16:49:10 dist-test-slave-t0tw krb5kdc[17091](info): commencing operation
krb5kdc: starting...
W20260928 16:49:11.023118 27028 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:11 dist-test-slave-t0tw krb5kdc[17091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:11.291229 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42319
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:11.643452 17104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:11.643792 17104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:11.643889 17104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:11.651460 17104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:11.651597 17104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:11.651641 17104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:11.651700 17104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:11.651743 17104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:11.662955 17104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:11.664525 17104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:11.666100 17104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:11.678579 17109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:11.678678 17112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:11.678566 17110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:11.679145 17104 server_base.cc:1034] running on GCE node
I20260928 16:49:11.680269 17104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:11.681913 17104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:11.683212 17104 hybrid_clock.cc:648] HybridClock initialized: now 1790614151683162 us; error 37 us; skew 500 ppm
I20260928 16:49:11.683624 17104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:11 dist-test-slave-t0tw krb5kdc[17091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614151, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:11.687300 17104 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:11.689049 17104 webserver.cc:460] Webserver started at http://127.26.101.62:46125/ using document root <none> and password file <none>
I20260928 16:49:11.689795 17104 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:11.689920 17104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:11.690239 17104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:11.693588 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "ff1b730c94a447bc8dcd5e573fc63d6d"
format_stamp: "Formatted at 2026-09-28 16:49:11 on dist-test-slave-t0tw"
server_key: "0a4267a5f57dc3ba1724cd1d6c388e80"
server_key_iv: "b82e4a3f3cad8fd2970d9f898146f87f"
server_key_version: "encryptionkey@0"
I20260928 16:49:11.694504 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "ff1b730c94a447bc8dcd5e573fc63d6d"
format_stamp: "Formatted at 2026-09-28 16:49:11 on dist-test-slave-t0tw"
server_key: "0a4267a5f57dc3ba1724cd1d6c388e80"
server_key_iv: "b82e4a3f3cad8fd2970d9f898146f87f"
server_key_version: "encryptionkey@0"
I20260928 16:49:11.700579 17104 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 16:49:11.704607 17119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:11.706118 17104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:49:11.706310 17104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "ff1b730c94a447bc8dcd5e573fc63d6d"
format_stamp: "Formatted at 2026-09-28 16:49:11 on dist-test-slave-t0tw"
server_key: "0a4267a5f57dc3ba1724cd1d6c388e80"
server_key_iv: "b82e4a3f3cad8fd2970d9f898146f87f"
server_key_version: "encryptionkey@0"
I20260928 16:49:11.706561 17104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:11.712759 17104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:11.717329 17104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:11.717610 17104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:11.735930 17104 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:42319
I20260928 16:49:11.735962 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:42319 every 8 connection(s)
I20260928 16:49:11.737334 17104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:11.741333 17171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:11.747714 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17104
I20260928 16:49:11.747915 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:11.748469 27028 external_mini_cluster.cc:1428] Setting key 20684d8fdf57e9903d0ee7374612a4aa
I20260928 16:49:11.753298 17171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d: Bootstrap starting.
I20260928 16:49:11.756809 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.750484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47927 (local address 127.26.101.62:42319)
0928 16:49:11.751503 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.751515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.751558 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:49:11.753271 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.753301 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.753704 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.754114 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:11.755209 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:11.755217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:11.755230 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:11.755245 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:11.755309 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:11.755334 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:11.755521 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.755620 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.755820 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":514,"threads_started":1}
I20260928 16:49:11.756956 17171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:11.758103 17171 log.cc:824] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:11.760682 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:11.761504 17171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d: No bootstrap required, opened a new log
I20260928 16:49:11.768406 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:11.765965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54337 (local address 127.26.101.62:42319)
0928 16:49:11.766162 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:11.766168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:11.766444 (+ 276us) server_negotiation.cc:407] Connection header received
0928 16:49:11.766619 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:11.766623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:11.766679 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:11.766767 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:11.767736 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:11.767740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:11.767743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:49:11.767759 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:11.767806 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:49:11.767825 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:11.767971 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:11.768080 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:49:11.768129 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:49:11.771217 17171 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } }
I20260928 16:49:11.771596 17171 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:49:11.771678 17171 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1b730c94a447bc8dcd5e573fc63d6d, State: Initialized, Role: FOLLOWER
I20260928 16:49:11.772235 17171 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } }
I20260928 16:49:11.772495 17171 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:49:11.772637 17171 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:49:11.772810 17171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:11.775182 17171 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } }
I20260928 16:49:11.775765 17171 leader_election.cc:302] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff1b730c94a447bc8dcd5e573fc63d6d; no voters:
I20260928 16:49:11.776401 17171 leader_election.cc:288] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:49:11.776520 17181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:11.777190 17181 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [term 1 LEADER]: Becoming Leader. State: Replica: ff1b730c94a447bc8dcd5e573fc63d6d, State: Running, Role: LEADER
I20260928 16:49:11.777727 17181 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } }
I20260928 16:49:11.778535 17171 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:11.780503 17182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } } }
I20260928 16:49:11.780977 17182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:11.780964 17183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1b730c94a447bc8dcd5e573fc63d6d. Latest consensus state: current_term: 1 leader_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1b730c94a447bc8dcd5e573fc63d6d" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 42319 } } }
I20260928 16:49:11.781817 17183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:11.782584 17187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:49:11.788024 17187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:11.798257 17187 catalog_manager.cc:1347] Generated new cluster ID: 14e6957a0f064b31acb76f754a516b21
I20260928 16:49:11.798400 17187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:11.816170 17187 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:11.816613 17187 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:11.827812 17187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1b730c94a447bc8dcd5e573fc63d6d: Generated new TSK 0
I20260928 16:49:11.828698 17187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:11.836062 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17104
W20260928 16:49:11.837239 17166 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47927
W20260928 16:49:11.838158 27028 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17104: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 16:49:11.838307 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17104
2026-09-28T16:49:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1846 ms)
[----------] 18 tests from AuthTokenIssuingTest (34186 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:11 dist-test-slave-t0tw krb5kdc[17203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:11 dist-test-slave-t0tw krb5kdc[17203](info): set up 2 sockets
Sep 28 16:49:11 dist-test-slave-t0tw krb5kdc[17203](info): commencing operation
krb5kdc: starting...
W20260928 16:49:12.883294 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:13 dist-test-slave-t0tw krb5kdc[17203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:13.155114 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:35045
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:13.496605 17216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:13.497008 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:13.497116 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:13.497229 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:13.504843 17216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:13.504969 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:13.505057 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:13.505126 17216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:13.505213 17216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:13.515789 17216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:35045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:13.517182 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:13.518553 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:13.530532 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:13.530498 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:13.530572 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:13.531745 17216 server_base.cc:1034] running on GCE node
I20260928 16:49:13.532554 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:13.534075 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:13.535303 17216 hybrid_clock.cc:648] HybridClock initialized: now 1790614153535231 us; error 44 us; skew 500 ppm
I20260928 16:49:13.535656 17216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:13 dist-test-slave-t0tw krb5kdc[17203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:13.539021 17216 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:13.540853 17216 webserver.cc:460] Webserver started at http://127.26.101.62:46757/ using document root <none> and password file <none>
I20260928 16:49:13.541666 17216 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:13.541846 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:13.542165 17216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:13.545480 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "152a1436dbf14000949635c00ea72b5c"
format_stamp: "Formatted at 2026-09-28 16:49:13 on dist-test-slave-t0tw"
server_key: "7d378440b464077fc6ffd9f148f586ba"
server_key_iv: "9e94f07f8e772e49e3074d9a54e14e2a"
server_key_version: "encryptionkey@0"
I20260928 16:49:13.546309 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "152a1436dbf14000949635c00ea72b5c"
format_stamp: "Formatted at 2026-09-28 16:49:13 on dist-test-slave-t0tw"
server_key: "7d378440b464077fc6ffd9f148f586ba"
server_key_iv: "9e94f07f8e772e49e3074d9a54e14e2a"
server_key_version: "encryptionkey@0"
I20260928 16:49:13.552078 17216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 16:49:13.556041 17231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:13.557643 17216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:49:13.557838 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "152a1436dbf14000949635c00ea72b5c"
format_stamp: "Formatted at 2026-09-28 16:49:13 on dist-test-slave-t0tw"
server_key: "7d378440b464077fc6ffd9f148f586ba"
server_key_iv: "9e94f07f8e772e49e3074d9a54e14e2a"
server_key_version: "encryptionkey@0"
I20260928 16:49:13.558046 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:13.573640 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:13.578334 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:13.578600 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:13.598102 17216 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:35045
I20260928 16:49:13.598412 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:35045 every 8 connection(s)
I20260928 16:49:13.599929 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:13.601512 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17216
I20260928 16:49:13.601753 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:13.602631 27028 external_mini_cluster.cc:1428] Setting key 571dae6a9e4e2d55ecd5f3db62dfac90
I20260928 16:49:13.605422 17283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:13.615008 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.26.101.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.61@KRBTEST.COM" created.
Sep 28 16:49:13 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:13.645089 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.618595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.26.101.62:35045)
0928 16:49:13.619579 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:13.619594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:13.620208 (+ 614us) server_negotiation.cc:407] Connection header received
0928 16:49:13.622383 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:13.622391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:13.622758 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:13.622987 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:13.624356 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:13.626156 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:13.627993 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:13.629036 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:13.629321 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:13.636583 (+ 7262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:13.636593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:13.636608 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:13.636648 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:13.639780 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:13.640651 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:13.640658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:13.640660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:13.640860 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:13.641284 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:13.641289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:13.641292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:13.641855 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:13.642077 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:13.642647 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 16:49:13.643006 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":718,"thread_start_us":435,"threads_started":1}
I20260928 16:49:13.645107 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.618595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35045 (local address 127.0.0.1:48116)
0928 16:49:13.619997 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.620046 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:13.621941 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:13.623031 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:13.623037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:13.623712 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:13.624115 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:13.624137 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:13.626387 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:13.626392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:13.627759 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:13.627777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:13.629365 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:13.629371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:13.629602 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:13.630511 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:13.630543 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:13.636239 (+ 5696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:13.639951 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:13.639965 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:13.639980 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:13.640502 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:13.640990 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:13.640996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:13.641000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:13.641169 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:13.642008 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:13.642017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:13.642252 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 16:49:13.642491 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 16:49:13.643174 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":205,"spinlock_wait_cycles":50048,"thread_start_us":858,"threads_started":1}
I20260928 16:49:13.648592 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:13.650032 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.649419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37912)
0928 16:49:13.649717 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.649865 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 16:49:13.650403 17235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
I20260928 16:49:13.652248 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.651777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37920)
0928 16:49:13.651960 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.652058 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:49:13.652483 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:33043: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
I20260928 16:49:13.670928 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:33043
--webserver_interface=127.26.101.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:13.701994 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } attempt: 1
I20260928 16:49:13.703421 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.702813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37922)
0928 16:49:13.703053 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.703232 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 16:49:13.704638 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.704202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37928)
0928 16:49:13.704363 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.704458 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:49:13.704869 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:33043: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
I20260928 16:49:13.773165 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } attempt: 2
I20260928 16:49:13.774664 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.774047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37932)
0928 16:49:13.774363 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.774487 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 16:49:13.775942 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.775501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37946)
0928 16:49:13.775667 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.775765 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:49:13.776209 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:33043: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
I20260928 16:49:13.867508 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } attempt: 3
I20260928 16:49:13.868845 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.868273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37958)
0928 16:49:13.868530 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.868665 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 16:49:13.870191 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:13.869727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37962)
0928 16:49:13.869900 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:13.869999 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 16:49:13.870419 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:33043: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
I20260928 16:49:14.013680 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } attempt: 4
I20260928 16:49:14.014923 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.014389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37976)
0928 16:49:14.014686 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.014777 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 16:49:14.015976 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.015611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37984)
0928 16:49:14.015735 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.015819 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:49:14.016230 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:33043: Network error: Client connection negotiation failed: client connection to 127.26.101.61:33043: connect: Connection refused (error 111)
W20260928 16:49:14.019029 17290 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:14.019439 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:14.019533 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:14.019655 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:14.027282 17290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:14.027395 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:14.027472 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:14.027546 17290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:14.027629 17290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:14.038043 17290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.101.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:14.039436 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:14.040805 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:14.052999 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:14.053031 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:14.052938 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:14.053879 17290 server_base.cc:1034] running on GCE node
I20260928 16:49:14.054718 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:14.056281 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:14.057515 17290 hybrid_clock.cc:648] HybridClock initialized: now 1790614154057451 us; error 37 us; skew 500 ppm
I20260928 16:49:14.057874 17290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:14.061280 17290 init.cc:377] Logged in from keytab as kudu/127.26.101.61@KRBTEST.COM (short username kudu)
I20260928 16:49:14.063040 17290 webserver.cc:460] Webserver started at http://127.26.101.61:44305/ using document root <none> and password file <none>
I20260928 16:49:14.063781 17290 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:14.063875 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:14.064158 17290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:14.067296 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data/instance:
uuid: "fff9d730dd754167bbb30ad0755bd448"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "7b8ab36dc0f095cc1235d9eeae579242"
server_key_iv: "a4bbdd0ab032620f57da147f4e0ab569"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.068120 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal/instance:
uuid: "fff9d730dd754167bbb30ad0755bd448"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "7b8ab36dc0f095cc1235d9eeae579242"
server_key_iv: "a4bbdd0ab032620f57da147f4e0ab569"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.073637 17290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 16:49:14.077243 17305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:14.078794 17290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:49:14.078984 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "fff9d730dd754167bbb30ad0755bd448"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "7b8ab36dc0f095cc1235d9eeae579242"
server_key_iv: "a4bbdd0ab032620f57da147f4e0ab569"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.079275 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:14.110158 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:14.114874 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:14.115190 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:14.134873 17290 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.61:33043
I20260928 16:49:14.134925 17356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.61:33043 every 8 connection(s)
I20260928 16:49:14.136344 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
I20260928 16:49:14.140532 17357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:14.141948 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17290
I20260928 16:49:14.142174 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-1/wal/instance
I20260928 16:49:14.142714 27028 external_mini_cluster.cc:1428] Setting key 51a09947eadabfe6381ff3c4847db868
I20260928 16:49:14.150228 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.60@KRBTEST.COM" created.
I20260928 16:49:14.177531 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.153952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.26.101.62:35045)
0928 16:49:14.154496 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.154502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.155780 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 16:49:14.158535 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.158543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.158710 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.158893 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.161130 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.162317 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.165455 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.165930 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.166053 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.170526 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.170532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.170535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.170587 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.173913 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.174856 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.174861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.174863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.175002 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.175523 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.175527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.175529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.175844 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.175971 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.176897 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.177146 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":274,"threads_started":1}
I20260928 16:49:14.178593 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.154684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35045 (local address 127.0.0.1:48130)
0928 16:49:14.155574 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.155624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.158088 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.159178 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.159191 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.159998 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.160821 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.160848 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.162687 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.162694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.164643 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.164672 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.166231 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.166239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.166569 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.167418 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.167450 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.170217 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.174110 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.174119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.174135 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.174697 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.175135 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.175140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.175143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.175400 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.175969 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.175975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.176446 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.176691 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.177159 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":365,"threads_started":1}
I20260928 16:49:14.180652 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:14.196924 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.181668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37988)
0928 16:49:14.181915 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.181934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.182037 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.183038 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.183043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.183552 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.183745 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.183755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.185205 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.185211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.186062 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.186074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.186941 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.186948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.187153 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.187756 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.187774 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.190006 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.194187 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.194192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.194195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.194637 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.195152 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.195156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.195159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.195321 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.196096 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.196100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.196233 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.196372 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.196618 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 16:49:14.197307 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.181806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.26.101.61:33043)
0928 16:49:14.182553 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.182560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.182587 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:49:14.182640 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.182647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.182868 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.183039 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.183922 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.185055 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.186197 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.186768 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.186884 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.190271 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.190288 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.190309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.190349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.193993 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.194799 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.194808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.194812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.194979 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.195442 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.195447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.195450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.195933 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.196134 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.196583 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.196872 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":419,"threads_started":1}
I20260928 16:49:14.201083 17357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:14.202585 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.202032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37634)
0928 16:49:14.202254 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.202409 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 16:49:14.202873 17306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.204566 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.204209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37640)
0928 16:49:14.204360 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.204447 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:49:14.204803 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
I20260928 16:49:14.214942 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:41021
--webserver_interface=127.26.101.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:14.254213 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 1
I20260928 16:49:14.255456 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.254980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37642)
0928 16:49:14.255196 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.255296 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:49:14.256484 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.256143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37648)
0928 16:49:14.256281 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.256374 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:49:14.256680 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.315462 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } attempt: 5
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:14.324942 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 2
I20260928 16:49:14.325981 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.325569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37650)
0928 16:49:14.325744 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.325836 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:49:14.326973 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.326627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37666)
0928 16:49:14.326775 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.326850 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:49:14.327286 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.330515 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.316255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:37996)
0928 16:49:14.316480 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.316501 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.316671 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.317076 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.317082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.317501 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.317693 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.317703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.318726 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.318731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.319654 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.319669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.320393 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.320399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.320594 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.321305 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.321325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.323493 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.328079 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.328083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.328087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.328518 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.328930 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.328933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.328936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.329083 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.329703 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.329707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.329870 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.330029 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.330280 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 16:49:14.330861 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.316351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.26.101.61:33043)
0928 16:49:14.316521 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.316527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.316596 (+ 69us) server_negotiation.cc:407] Connection header received
0928 16:49:14.316836 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.316840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.316926 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.317011 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.317873 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.318577 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.319802 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.320226 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.320339 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.323714 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.323781 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.323784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.323823 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.327926 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.328642 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.328648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.328652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.328810 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.329180 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.329185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.329188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.329555 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.329705 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.330154 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.330386 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:49:14.331893 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:14.333437 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.332942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37670)
0928 16:49:14.333139 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.333255 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:49:14.334545 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.334107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37682)
0928 16:49:14.334364 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.334436 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260928 16:49:14.334743 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.385968 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 1
I20260928 16:49:14.387230 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.386718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37696)
0928 16:49:14.386919 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.387011 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:49:14.388298 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.387900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37712)
0928 16:49:14.388083 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.388164 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 16:49:14.388562 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.418516 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 3
I20260928 16:49:14.419786 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.419297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37726)
0928 16:49:14.419511 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.419612 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 16:49:14.420876 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.420451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37742)
0928 16:49:14.420668 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.420748 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 16:49:14.421088 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.456800 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 2
I20260928 16:49:14.458163 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.457614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37752)
0928 16:49:14.457894 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.457990 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 16:49:14.459219 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.458844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37758)
0928 16:49:14.458992 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.459065 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:49:14.459445 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
W20260928 16:49:14.563148 17365 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:14.563565 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:14.563660 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:14.563772 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 16:49:14.564307 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 4
I20260928 16:49:14.565436 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.564938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37770)
0928 16:49:14.565169 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.565258 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 16:49:14.565641 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 3
I20260928 16:49:14.566746 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.566253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37786)
0928 16:49:14.566434 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.566534 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:49:14.567137 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.566631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37792)
0928 16:49:14.566843 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.566938 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 16:49:14.567406 17357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
I20260928 16:49:14.567960 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.567495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:37800)
0928 16:49:14.567706 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.567805 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 16:49:14.568204 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:41021: Network error: Client connection negotiation failed: client connection to 127.26.101.60:41021: connect: Connection refused (error 111)
W20260928 16:49:14.572441 17365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:14.572552 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:14.572611 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:14.572654 17365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:14.572716 17365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:14.583204 17365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:35045,127.26.101.61:33043,127.26.101.60:41021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.101.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:14.584617 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:14.585983 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:14.597582 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:14.597683 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:14.598129 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:14.598924 17365 server_base.cc:1034] running on GCE node
I20260928 16:49:14.599759 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:14.601351 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:14.602582 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790614154602501 us; error 61 us; skew 500 ppm
I20260928 16:49:14.602916 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:14.606168 17365 init.cc:377] Logged in from keytab as kudu/127.26.101.60@KRBTEST.COM (short username kudu)
I20260928 16:49:14.607815 17365 webserver.cc:460] Webserver started at http://127.26.101.60:37937/ using document root <none> and password file <none>
I20260928 16:49:14.608551 17365 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:14.608647 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:14.608981 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:14.612190 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data/instance:
uuid: "3801a4684d134159b11edfa5c9ae7777"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "5d273dcdf19a33d82d05f982fee9f5a1"
server_key_iv: "c2f74fb5371b43e6fb187f7be132b1a4"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.613018 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal/instance:
uuid: "3801a4684d134159b11edfa5c9ae7777"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "5d273dcdf19a33d82d05f982fee9f5a1"
server_key_iv: "c2f74fb5371b43e6fb187f7be132b1a4"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.618471 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 16:49:14.622083 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:14.623522 17365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:49:14.623674 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "3801a4684d134159b11edfa5c9ae7777"
format_stamp: "Formatted at 2026-09-28 16:49:14 on dist-test-slave-t0tw"
server_key: "5d273dcdf19a33d82d05f982fee9f5a1"
server_key_iv: "c2f74fb5371b43e6fb187f7be132b1a4"
server_key_version: "encryptionkey@0"
I20260928 16:49:14.623915 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:14.635653 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:14.640050 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:14.640354 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:14.658687 17365 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.60:41021
I20260928 16:49:14.658694 17431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.60:41021 every 8 connection(s)
I20260928 16:49:14.660204 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
I20260928 16:49:14.661675 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17365
I20260928 16:49:14.661845 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613789516984-27028-0/minicluster-data/master-2/wal/instance
I20260928 16:49:14.662264 27028 external_mini_cluster.cc:1428] Setting key 770d17e7dbb019f2072fd3a8d4c3df8b
I20260928 16:49:14.664846 17432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:14.672142 17432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:14.685035 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.664399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.26.101.60:41021)
0928 16:49:14.665192 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.665204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.665245 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:49:14.666951 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.666974 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.667428 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.667814 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.669286 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.670628 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.671977 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.672681 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.672852 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.677530 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.677537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.677548 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.677584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.680536 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.681238 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.681244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.681251 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.681477 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.681851 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.681854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.681856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.682380 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.682668 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.683244 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.683666 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":415,"threads_started":1}
I20260928 16:49:14.689913 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.674844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:35045 (local address 127.0.0.1:37928)
0928 16:49:14.675612 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.675655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.675889 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.676310 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.676318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.676993 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.677247 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.677257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.678451 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.678455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.679680 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.679700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.680473 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.680480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.680643 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.681230 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.681257 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.683520 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.687031 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.687043 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.687050 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.687495 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.687925 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.687928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.687931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.688097 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.688817 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.688823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.689022 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.689168 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.689502 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":411,"threads_started":1}
I20260928 16:49:14.689906 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.674820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.26.101.62:35045)
0928 16:49:14.674989 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.674993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.675756 (+ 763us) server_negotiation.cc:407] Connection header received
0928 16:49:14.676050 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.676055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.676135 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.676215 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.677466 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.678285 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.679848 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.680281 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.680398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.683737 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.683743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.683748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.683790 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.686901 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.687629 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.687634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.687637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.687797 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.688238 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.688243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.688245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.688595 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.688743 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.689354 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.689592 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:49:14.691311 17432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:14.704336 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.688383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.26.101.62:35045)
0928 16:49:14.688976 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.688980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.689001 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:49:14.689042 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.689045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.689099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.689193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.690569 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.691474 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.693002 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.693465 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.693591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.698044 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.698053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.698059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.698107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.701254 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.702128 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.702132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.702135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.702303 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.702723 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.702729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.702732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.703187 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.703513 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.703814 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.704000 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":278,"threads_started":1}
I20260928 16:49:14.705440 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.692065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:33043 (local address 127.0.0.1:49634)
0928 16:49:14.692256 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.692277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.692395 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.692670 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.692675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.693108 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.693246 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.693254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.694308 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.694313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.695244 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.695257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.695973 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.695979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.696147 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.696662 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.696679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.699306 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.702360 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.702365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.702369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.702936 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.703429 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.703434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.703438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.703632 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.704348 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.704353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.704642 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.704780 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.705094 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 16:49:14.705477 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.692079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.26.101.61:33043)
0928 16:49:14.692258 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.692264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.692338 (+ 74us) server_negotiation.cc:407] Connection header received
0928 16:49:14.692491 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.692495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.692555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.692631 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.693393 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.694167 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.695383 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.695790 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.695903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.699504 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.699509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.699512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.699547 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.702216 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.703061 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.703065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.703067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.703287 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.703882 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.703887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.703890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.704229 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.704391 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.704893 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.705115 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:49:14.706387 17432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:14.719492 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.707117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:51770)
0928 16:49:14.707275 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.707294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.707399 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.707670 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.707678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.708122 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.708252 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.708260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.709154 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.709159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.709995 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.710006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.710654 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.710660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.710838 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.711440 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.711456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.713558 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.716973 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.716977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.716979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.717330 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.717743 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.717746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.717749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.717876 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.718442 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.718448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.718623 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.718779 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.719055 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 16:49:14.719588 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.707158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.26.101.60:41021)
0928 16:49:14.707324 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.707328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.707343 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:49:14.707497 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.707500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.707551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.707623 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.708378 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.709042 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.710117 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.710513 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.710613 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.713760 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.713766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.713771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.713818 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.716808 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.717458 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.717463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.717467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.717622 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.717995 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.717999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.718002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.718312 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.718420 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.718936 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.719234 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":57}
I20260928 16:49:14.720752 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.706145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.26.101.61:33043)
0928 16:49:14.706275 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.706287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.706301 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:49:14.706338 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.706341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.706393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.706464 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.708017 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.708736 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.709911 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.710342 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.710451 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.714905 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.714912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.714916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.714957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.717978 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.718727 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.718731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.718733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.718909 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.719377 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.719382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.719386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.719726 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.719999 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.720307 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.720491 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:49:14.725641 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: Bootstrap starting.
I20260928 16:49:14.726267 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:14.729882 17432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:14.730983 17432 log.cc:824] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:14.729944 17258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "152a1436dbf14000949635c00ea72b5c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37938
I20260928 16:49:14.734109 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: No bootstrap required, opened a new log
I20260928 16:49:14.741418 17432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.741793 17432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3801a4684d134159b11edfa5c9ae7777, State: Initialized, Role: FOLLOWER
I20260928 16:49:14.742257 17432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.743242 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.743621 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:14.744233 17432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:14.745428 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 4
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614153, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
W20260928 16:49:14.754911 17450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:14.755044 17450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:14.752372 17365 master_runner.cc:186] Error getting master registration for 127.26.101.62:35045: OK, instance_id { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" instance_seqno: 1790614153592509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:14.760707 17365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:14.763912 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.746119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:51776)
0928 16:49:14.746338 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.746361 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.746501 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.746888 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.746892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.747506 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.747715 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.747731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.748948 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.748954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.750347 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.750367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.751361 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.751369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.751598 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.752295 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.752318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.755296 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.760885 (+ 5589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.760891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.760896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.761485 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.761931 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.761936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.761940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.762134 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.762823 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.762829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.763018 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.763192 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.763524 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:49:14.763922 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.746286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.26.101.60:41021)
0928 16:49:14.746506 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.746513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.746538 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:49:14.746628 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.746633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.746719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.746846 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.747892 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.748803 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.750493 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.751184 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.751313 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.755523 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.755532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.755537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.755577 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.760642 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.761652 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.761663 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.761667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.761818 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.762287 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.762293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.762296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.762697 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.762886 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.763327 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.763560 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:49:14.769893 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: Bootstrap starting.
I20260928 16:49:14.774152 17283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:14.775523 17283 log.cc:824] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:14.778669 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: No bootstrap required, opened a new log
I20260928 16:49:14.784704 17283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.785158 17283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152a1436dbf14000949635c00ea72b5c, State: Initialized, Role: FOLLOWER
I20260928 16:49:14.785678 17283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.786701 17453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.787199 17453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:14.788043 17283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:14.788156 17258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:49:14.788332 17258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:14.792678 17258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.794533 17258 leader_election.cc:288] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [CANDIDATE]: Term 1 election: Requested vote from peers fff9d730dd754167bbb30ad0755bd448 (127.26.101.61:33043), 3801a4684d134159b11edfa5c9ae7777 (127.26.101.60:41021)
I20260928 16:49:14.795707 17407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152a1436dbf14000949635c00ea72b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3801a4684d134159b11edfa5c9ae7777"
I20260928 16:49:14.796280 17407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:14.800341 17407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152a1436dbf14000949635c00ea72b5c in term 1.
I20260928 16:49:14.801373 17232 leader_election.cc:302] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152a1436dbf14000949635c00ea72b5c, 3801a4684d134159b11edfa5c9ae7777; no voters:
I20260928 16:49:14.801991 17453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:14.806000 17453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [term 1 LEADER]: Becoming Leader. State: Replica: 152a1436dbf14000949635c00ea72b5c, State: Running, Role: LEADER
I20260928 16:49:14.806734 17453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.818296 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152a1436dbf14000949635c00ea72b5c. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.818730 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:14.819950 17466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 16:49:14.832438 17237 tablet.cc:2369] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:14.841603 17466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:14.866796 17357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } attempt: 5
I20260928 16:49:14.883695 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.832010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.26.101.60:41021)
0928 16:49:14.832199 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.832206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.845265 (+ 13059us) server_negotiation.cc:407] Connection header received
0928 16:49:14.847435 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.847441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.847537 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.847621 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.852943 (+ 5322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.854098 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.859562 (+ 5464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.860324 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.860477 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.864674 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.864682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.864688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.864731 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.870613 (+ 5882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.880084 (+ 9471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.880090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.880094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.880289 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.881134 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.881144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.881148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.881581 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.881925 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.883012 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.883313 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 16:49:14 dist-test-slave-t0tw krb5kdc[17203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614154, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
W20260928 16:49:14.892622 17386 tablet.cc:2369] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:14.893814 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.842500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.26.101.61:33043)
0928 16:49:14.842684 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.842690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.849566 (+ 6876us) server_negotiation.cc:407] Connection header received
0928 16:49:14.850743 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.850749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.850834 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.850906 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.860116 (+ 9210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.861018 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.869251 (+ 8233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.869903 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.870009 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.879561 (+ 9552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.879570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.879575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.879634 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.883395 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.884300 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.884316 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.884319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.884503 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.890302 (+ 5799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.890319 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.890323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.890930 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.891237 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.891914 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.892162 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":23936}
I20260928 16:49:14.898489 17407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [term 1 FOLLOWER]: Refusing update from remote peer 152a1436dbf14000949635c00ea72b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:14.899887 17453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:14.918228 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.867522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:41021 (local address 127.0.0.1:51794)
0928 16:49:14.868025 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:14.868044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:14.868150 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:14.884616 (+ 16466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:14.884621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:14.885113 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:14.885326 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.885339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.886912 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.886918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.888310 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:14.888328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.889398 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.889404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:14.889659 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.890306 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:14.890330 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:14.893271 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:14.910387 (+ 17116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.910393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.910397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.911400 (+ 1003us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.913661 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:14.913666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:14.913670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:14.913970 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:14.916788 (+ 2818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:14.916793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:14.916984 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 16:49:14.917235 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 16:49:14.917589 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372}
I20260928 16:49:14.919674 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.868360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.26.101.60:41021)
0928 16:49:14.884292 (+ 15932us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.884297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.884312 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:49:14.884361 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.884371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.884454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.885476 (+ 1022us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.885624 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.886743 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.888515 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.889210 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.890037 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.893492 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.893498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.893503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.893541 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.909653 (+ 16112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.912109 (+ 2456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.912116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.912120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.912310 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.914169 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.914174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.914178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.914591 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.918696 (+ 4105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.918858 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.919224 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15819,"threads_started":1}
I20260928 16:49:14.921237 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:14.838486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.26.101.62:35045)
0928 16:49:14.838672 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:14.838679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:14.846879 (+ 8200us) server_negotiation.cc:407] Connection header received
0928 16:49:14.848840 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:14.848846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:14.848935 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:14.849011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:14.854327 (+ 5316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.855368 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.859618 (+ 4250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:14.861933 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:14.862069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:14.899059 (+ 36990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:14.899067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:14.899071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:14.899137 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:14.903333 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.912837 (+ 9504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.912844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.912848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.913051 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:14.915120 (+ 2069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:14.915126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:14.915130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:14.915671 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:14.915986 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:14.920457 (+ 4471us) server_negotiation.cc:299] Negotiation successful
0928 16:49:14.920780 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:49:14.915469 17216 master_runner.cc:186] Error getting master registration for 127.26.101.61:33043: OK, instance_id { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" instance_seqno: 1790614154129258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:14.929777 17216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:14.932348 17357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: Bootstrap starting.
I20260928 16:49:14.937043 17357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:14.938704 17357 log.cc:824] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:14.940833 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152a1436dbf14000949635c00ea72b5c. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.941157 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:14.942131 17475 mvcc.cc:205] Tried to move back new op lower bound from 7334355578380185600 to 7334355578096029696. Current Snapshot: MvccSnapshot[applied={T|T < 7334355578380185600}]
I20260928 16:49:14.944497 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.944779 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:14.946661 17357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: No bootstrap required, opened a new log
I20260928 16:49:14.960474 17466 catalog_manager.cc:1347] Generated new cluster ID: 3512ce91ed324674a1a9e30f8d137bc0
I20260928 16:49:14.960613 17466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:14.962121 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.962426 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:14.964044 17357 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.964875 17357 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff9d730dd754167bbb30ad0755bd448, State: Initialized, Role: FOLLOWER
I20260928 16:49:14.965839 17357 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } }
I20260928 16:49:14.977918 17332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152a1436dbf14000949635c00ea72b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fff9d730dd754167bbb30ad0755bd448"
I20260928 16:49:14.978415 17331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:14.978667 17357 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:14.983572 17331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [term 1 FOLLOWER]: Refusing update from remote peer 152a1436dbf14000949635c00ea72b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:14.984808 17453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:14.987375 17332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152a1436dbf14000949635c00ea72b5c in term 1.
I20260928 16:49:14.989970 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:14.990526 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:15.014410 17466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:15.016425 17466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:15.024343 17485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:15.024886 17485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:15.024371 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152a1436dbf14000949635c00ea72b5c. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:15.025399 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:15.026708 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "152a1436dbf14000949635c00ea72b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152a1436dbf14000949635c00ea72b5c" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 35045 } } peers { permanent_uuid: "fff9d730dd754167bbb30ad0755bd448" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 33043 } } peers { permanent_uuid: "3801a4684d134159b11edfa5c9ae7777" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 41021 } } }
I20260928 16:49:15.026993 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:15.041606 17486 mvcc.cc:205] Tried to move back new op lower bound from 7334355578380185600 to 7334355578096029696. Current Snapshot: MvccSnapshot[applied={T|T < 7334355578380185600 or (T in {7334355578380185600})}]
I20260928 16:49:15.063398 17466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 152a1436dbf14000949635c00ea72b5c: Generated new TSK 0
I20260928 16:49:15.068958 17466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:15.111204 17498 catalog_manager.cc:1259] Loaded cluster ID: 3512ce91ed324674a1a9e30f8d137bc0
I20260928 16:49:15.111419 17498 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: loading cluster ID for follower catalog manager: success
I20260928 16:49:15.115448 17498 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: acquiring CA information for follower catalog manager: success
I20260928 16:49:15.118345 17498 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fff9d730dd754167bbb30ad0755bd448: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:49:15.222783 17437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.202244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.26.101.62:35045)
0928 16:49:15.202464 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.202476 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.219652 (+ 17176us) server_negotiation.cc:407] Connection header received
0928 16:49:15.221063 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.221070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.221163 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.221240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.222470 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37962: BlockingRecv error: recv got EOF from 127.0.0.1:37962 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 16:49:15.228837 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.204470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.26.101.60:41021)
0928 16:49:15.204691 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.204698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.225299 (+ 20601us) server_negotiation.cc:407] Connection header received
0928 16:49:15.226535 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.226541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.226692 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.226771 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.228500 (+ 1729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51804: BlockingRecv error: recv got EOF from 127.0.0.1:51804 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:49:15.229586 17362 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.217641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.26.101.61:33043)
0928 16:49:15.217876 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.217882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.227721 (+ 9839us) server_negotiation.cc:407] Connection header received
0928 16:49:15.227978 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.227984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.228083 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.228156 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.229366 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49666: BlockingRecv error: recv got EOF from 127.0.0.1:49666 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 16:49:15.286259 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.260941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.26.101.61:33043)
0928 16:49:15.261139 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.261146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.277996 (+ 16850us) server_negotiation.cc:407] Connection header received
0928 16:49:15.278373 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.278380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.278617 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.278774 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.279577 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.280803 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.282962 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.283725 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.283835 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.284686 (+ 851us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.285414 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.285529 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.285797 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.285900 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:49:15.311774 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.309319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.26.101.60:41021)
0928 16:49:15.309515 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.309522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.310058 (+ 536us) server_negotiation.cc:407] Connection header received
0928 16:49:15.310272 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.310278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.310415 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.310549 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.311526 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51816: BlockingRecv error: recv got EOF from 127.0.0.1:51816 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 16:49:15.324275 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.317159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.26.101.61:33043)
0928 16:49:15.317368 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.317374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.317933 (+ 559us) server_negotiation.cc:407] Connection header received
0928 16:49:15.318163 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.318170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.318268 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.318383 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.318910 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.320116 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.322040 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.322762 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.322918 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.323424 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.323602 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.323705 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.323865 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.323961 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 16:49:15.340608 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.336917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.26.101.60:41021)
0928 16:49:15.337111 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.337117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.338963 (+ 1846us) server_negotiation.cc:407] Connection header received
0928 16:49:15.339214 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.339221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.339339 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.339466 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.340374 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51820: BlockingRecv error: recv got EOF from 127.0.0.1:51820 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:49:15.363289 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.353157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.26.101.61:33043)
0928 16:49:15.353347 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.353354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.356607 (+ 3253us) server_negotiation.cc:407] Connection header received
0928 16:49:15.356823 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.356830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.356931 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.357052 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.357502 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.358716 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.360744 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.361486 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.361634 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.362261 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.362445 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.362549 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.362735 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.362845 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 16:49:15.377249 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.372972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.26.101.60:41021)
0928 16:49:15.373160 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.373166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.375682 (+ 2516us) server_negotiation.cc:407] Connection header received
0928 16:49:15.375876 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.375884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.375983 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.376100 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.376988 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51826: BlockingRecv error: recv got EOF from 127.0.0.1:51826 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:49:15.394147 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.384758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.26.101.61:33043)
0928 16:49:15.385012 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.385019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.387713 (+ 2694us) server_negotiation.cc:407] Connection header received
0928 16:49:15.387924 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.387931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.388029 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.388122 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.388707 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.389871 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.391946 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.392630 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.392773 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.393291 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.393431 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.393524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.393745 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.393857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260928 16:49:15.407706 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.405388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.26.101.60:41021)
0928 16:49:15.405623 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.405629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.406181 (+ 552us) server_negotiation.cc:407] Connection header received
0928 16:49:15.406361 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.406366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.406464 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.406574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.407488 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51838: BlockingRecv error: recv got EOF from 127.0.0.1:51838 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 16:49:15.427917 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.420843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.26.101.61:33043)
0928 16:49:15.421037 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.421045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.421583 (+ 538us) server_negotiation.cc:407] Connection header received
0928 16:49:15.421791 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.421796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.421883 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.421976 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.422476 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.423571 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.425619 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.426312 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.426458 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.426953 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.427170 (+ 217us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.427271 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.427439 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.427536 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:49:15.443367 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.441081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.26.101.60:41021)
0928 16:49:15.441273 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.441278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.441804 (+ 526us) server_negotiation.cc:407] Connection header received
0928 16:49:15.442004 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.442010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.442108 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.442216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.443126 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51852: BlockingRecv error: recv got EOF from 127.0.0.1:51852 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:49:15.470455 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.463530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.26.101.61:33043)
0928 16:49:15.463735 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.463744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.464273 (+ 529us) server_negotiation.cc:407] Connection header received
0928 16:49:15.464475 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.464482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.464602 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.464705 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.465188 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.466415 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.468245 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.468940 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.469081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.469588 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.469764 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.469858 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.470077 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.470172 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 16:49:15.478977 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.476953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.26.101.60:41021)
0928 16:49:15.477138 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.477144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.477531 (+ 387us) server_negotiation.cc:407] Connection header received
0928 16:49:15.477707 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.477725 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.477802 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.477920 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.478707 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51860: BlockingRecv error: recv got EOF from 127.0.0.1:51860 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:49:15.520884 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.514793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.26.101.61:33043)
0928 16:49:15.515101 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.515108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.515425 (+ 317us) server_negotiation.cc:407] Connection header received
0928 16:49:15.515584 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.515593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.515713 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.515843 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.516282 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.517426 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.518792 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.519564 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.519703 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.520100 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.520242 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.520333 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.520447 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.520533 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 16:49:15.528980 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.526859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.26.101.60:41021)
0928 16:49:15.527111 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.527117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.527476 (+ 359us) server_negotiation.cc:407] Connection header received
0928 16:49:15.527686 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.527690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.527765 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.527845 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.528680 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51868: BlockingRecv error: recv got EOF from 127.0.0.1:51868 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 16:49:15.602550 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.596621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.26.101.61:33043)
0928 16:49:15.596789 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.596794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.597157 (+ 363us) server_negotiation.cc:407] Connection header received
0928 16:49:15.597294 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.597298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.597375 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.597471 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.598006 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.598742 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.600772 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.601224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.601333 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.601855 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.601979 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.602049 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.602205 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.602280 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 16:49:15.611618 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.609328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.26.101.60:41021)
0928 16:49:15.609541 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.609545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.609970 (+ 425us) server_negotiation.cc:407] Connection header received
0928 16:49:15.610206 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.610212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.610299 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.610408 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.611375 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51880: BlockingRecv error: recv got EOF from 127.0.0.1:51880 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 16:49:15.749202 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:15.743355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.26.101.61:33043)
0928 16:49:15.743563 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.743571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.743972 (+ 401us) server_negotiation.cc:407] Connection header received
0928 16:49:15.744199 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.744204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.744275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.744388 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:15.744820 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.745594 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.747198 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:15.747693 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:15.747816 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:15.748284 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:15.748508 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:15.748601 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:15.748777 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 16:49:15.748870 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:49:15.757190 17435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:15.755168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.26.101.60:41021)
0928 16:49:15.755407 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:15.755413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:15.755773 (+ 360us) server_negotiation.cc:407] Connection header received
0928 16:49:15.755937 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:15.755941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:15.756010 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:15.756098 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:15.756962 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51894: BlockingRecv error: recv got EOF from 127.0.0.1:51894 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 16:49:15.757494 17450 catalog_manager.cc:1259] Loaded cluster ID: 3512ce91ed324674a1a9e30f8d137bc0
I20260928 16:49:15.757586 17450 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: loading cluster ID for follower catalog manager: success
I20260928 16:49:15.760154 17450 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: acquiring CA information for follower catalog manager: success
I20260928 16:49:15.762012 17450 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3801a4684d134159b11edfa5c9ae7777: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:16.026268 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:16.021053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.26.101.61:33043)
0928 16:49:16.021303 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:16.021310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:16.021499 (+ 189us) server_negotiation.cc:407] Connection header received
0928 16:49:16.021735 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:16.021739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:16.021823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:16.021904 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:16.022394 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.023187 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.024553 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.024995 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.025111 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:16.025575 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:16.025687 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:16.025773 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:16.025913 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:49:16.026000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 16:49:16.037853 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:16.031908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.26.101.60:41021)
0928 16:49:16.032119 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:16.032125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:16.032320 (+ 195us) server_negotiation.cc:407] Connection header received
0928 16:49:16.032521 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:16.032526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:16.032677 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:16.032783 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:16.033213 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.033942 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.035322 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.035872 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.036016 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:16.036589 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:16.037176 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:16.037338 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:16.037444 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 16:49:16.037538 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:49:16.052760 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:16.044311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.26.101.62:35045)
0928 16:49:16.045969 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:16.045976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:16.045999 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:49:16.046152 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:16.046158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:16.046475 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:16.047316 (+ 841us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:16.047515 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.048351 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.050203 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.050610 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.050724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:16.051371 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:16.051853 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:16.051968 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:16.052119 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 16:49:16.052217 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1383,"threads_started":1}
I20260928 16:49:16.053496 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:16.044683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.26.101.60:41021)
0928 16:49:16.044889 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:16.044895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:16.047058 (+ 2163us) server_negotiation.cc:407] Connection header received
0928 16:49:16.047316 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:16.047321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:16.047420 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:16.047511 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:16.048271 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.049031 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.051347 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.051800 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.051874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:16.052490 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:16.052592 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:16.052668 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:16.053154 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 16:49:16.053227 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 16:49:16.053893 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:16.044358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.26.101.61:33043)
0928 16:49:16.044580 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:16.044586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:16.046840 (+ 2254us) server_negotiation.cc:407] Connection header received
0928 16:49:16.047167 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:16.047174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:16.047266 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:16.047326 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:16.048302 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.049338 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.051471 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:16.052062 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:16.052178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:16.052748 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:16.052876 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:16.052967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:16.053504 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 16:49:16.053587 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:49:16.058490 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17216
I20260928 16:49:16.059592 17278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:16.172734 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17216
W20260928 16:49:16.191818 17307 connection.cc:535] server connection from 127.0.0.1:37996 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:49:16.192283 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17290
I20260928 16:49:16.193499 17352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:16.300350 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17290
I20260928 16:49:16.318209 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17365
I20260928 16:49:16.319350 17427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:16.426793 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17365
2026-09-28T16:49:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4592 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:49:16 dist-test-slave-t0tw krb5kdc[17623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:49:16 dist-test-slave-t0tw krb5kdc[17623](info): set up 2 sockets
Sep 28 16:49:16 dist-test-slave-t0tw krb5kdc[17623](info): commencing operation
krb5kdc: starting...
W20260928 16:49:17.483422 27028 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/test-user.keytab.
Sep 28 16:49:17 dist-test-slave-t0tw krb5kdc[17623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.26.101.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
Entry for principal kudu/127.26.101.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab.
I20260928 16:49:17.758548 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37943
--webserver_interface=127.26.101.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:49:18.101905 17636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:18.102331 17636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:18.102444 17636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:18.102564 17636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:18.110150 17636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:18.110280 17636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:18.110365 17636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:18.110440 17636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:18.110526 17636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:18.121027 17636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.62.keytab
--principal=kudu/127.26.101.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.62:37943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.101.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:18.122464 17636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:18.123840 17636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:18.135553 17641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:18.135512 17642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:18.135576 17644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:18.136924 17636 server_base.cc:1034] running on GCE node
I20260928 16:49:18.137776 17636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:18.139394 17636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:18.140632 17636 hybrid_clock.cc:648] HybridClock initialized: now 1790614158140559 us; error 41 us; skew 500 ppm
I20260928 16:49:18.141042 17636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:18.144376 17636 init.cc:377] Logged in from keytab as kudu/127.26.101.62@KRBTEST.COM (short username kudu)
I20260928 16:49:18.146039 17636 webserver.cc:460] Webserver started at http://127.26.101.62:37461/ using document root <none> and password file <none>
I20260928 16:49:18.146777 17636 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:18.146871 17636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:18.147262 17636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:18.150405 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data/instance:
uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "6e53a1ccc2bccdfbe0ee0173dc9310fe"
server_key_iv: "0c5eed728dbbb0e1b94aeab4507bf2a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.151314 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance:
uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "6e53a1ccc2bccdfbe0ee0173dc9310fe"
server_key_iv: "0c5eed728dbbb0e1b94aeab4507bf2a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.157779 17636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 16:49:18.163221 17651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:18.165004 17636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:49:18.165210 17636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal
uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "6e53a1ccc2bccdfbe0ee0173dc9310fe"
server_key_iv: "0c5eed728dbbb0e1b94aeab4507bf2a9"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.165441 17636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:18.185097 17636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:18.189472 17636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:18.189754 17636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:18.207877 17636 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.62:37943
I20260928 16:49:18.207898 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.62:37943 every 8 connection(s)
I20260928 16:49:18.209285 17636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/data/info.pb
I20260928 16:49:18.213339 17703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:18.215724 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17636
I20260928 16:49:18.215919 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-0/wal/instance
I20260928 16:49:18.216483 27028 external_mini_cluster.cc:1428] Setting key 44798be6e896e7d1cac42b59f6b93ad4
I20260928 16:49:18.222508 17703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.26.101.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.61@KRBTEST.COM" created.
I20260928 16:49:18.247843 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.226570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37943 (local address 127.0.0.1:57664)
0928 16:49:18.227601 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.227650 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:18.230096 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:18.231199 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:18.231206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:18.231886 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:18.232248 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.232267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.234365 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.234369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.235642 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.235657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.236965 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.236973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.237116 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.237801 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:18.237825 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:18.240126 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:18.243699 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.243711 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.243726 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.244138 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.244562 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.244567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.244579 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.244748 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.245671 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:18.245678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:18.245886 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 16:49:18.246110 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 16:49:18.246601 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":458,"threads_started":1}
I20260928 16:49:18.247900 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.226570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.26.101.62:37943)
0928 16:49:18.228542 (+ 1972us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:18.228549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:18.228595 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:49:18.230525 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:18.230533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:18.230869 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:18.231101 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:18.232483 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.234173 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.235814 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.236671 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.236844 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.240424 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:18.240435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:18.240453 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:18.240491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:18.243557 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.244291 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.244298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.244300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.244451 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.244898 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.244903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.244905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.245484 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:18.245662 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:18.246327 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 16:49:18.246738 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":1774,"spinlock_wait_cycles":24320,"thread_start_us":1500,"threads_started":1}
I20260928 16:49:18.251207 17703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:18.252242 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.251844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37170)
0928 16:49:18.252012 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.252125 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 16:49:18.252475 17654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
I20260928 16:49:18.254026 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.253670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37186)
0928 16:49:18.253814 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.253885 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:49:18.254205 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:37261: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
Entry for principal kudu/127.26.101.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab.
I20260928 16:49:18.278596 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:37261
--webserver_interface=127.26.101.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:18.303545 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } attempt: 1
I20260928 16:49:18.304904 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.304334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37198)
0928 16:49:18.304568 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.304705 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 16:49:18.305948 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.305549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37204)
0928 16:49:18.305690 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.305777 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:49:18.306135 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:37261: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
I20260928 16:49:18.374403 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } attempt: 2
I20260928 16:49:18.375756 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.375186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37220)
0928 16:49:18.375447 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.375561 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 16:49:18.376987 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.376490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37226)
0928 16:49:18.376651 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.376751 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:49:18.377245 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:37261: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
I20260928 16:49:18.468544 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } attempt: 3
I20260928 16:49:18.469875 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.469303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37232)
0928 16:49:18.469583 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.469701 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 16:49:18.471107 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.470622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37246)
0928 16:49:18.470797 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.470896 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 16:49:18.471304 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:37261: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
I20260928 16:49:18.614558 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } attempt: 4
I20260928 16:49:18.615887 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.615296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37256)
0928 16:49:18.615613 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.615721 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 16:49:18.616902 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.616522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37272)
0928 16:49:18.616663 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.616760 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:49:18.617097 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.61:37261: Network error: Client connection negotiation failed: client connection to 127.26.101.61:37261: connect: Connection refused (error 111)
W20260928 16:49:18.618180 17710 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:18.618505 17710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:18.618597 17710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:18.618706 17710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:18.626605 17710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:18.626718 17710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:18.626807 17710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:18.626864 17710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:18.626946 17710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:18.637153 17710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.61.keytab
--principal=kudu/127.26.101.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.61:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.101.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:18.638474 17710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:18.639840 17710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:18.651804 17715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:18.651834 17716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:18.651865 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:18.652697 17710 server_base.cc:1034] running on GCE node
I20260928 16:49:18.653574 17710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:18.655022 17710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:18.656301 17710 hybrid_clock.cc:648] HybridClock initialized: now 1790614158656225 us; error 47 us; skew 500 ppm
I20260928 16:49:18.656648 17710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:18.659967 17710 init.cc:377] Logged in from keytab as kudu/127.26.101.61@KRBTEST.COM (short username kudu)
I20260928 16:49:18.661476 17710 webserver.cc:460] Webserver started at http://127.26.101.61:37071/ using document root <none> and password file <none>
I20260928 16:49:18.662266 17710 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:18.662385 17710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:18.662674 17710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:18.665861 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data/instance:
uuid: "b2bd236f82de4f98abce9dd4b8639411"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "1ae696bd43e904bcd05e3c9f2a4b0d00"
server_key_iv: "bde7eb80203fea94dd0394c4ea1839fb"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.666649 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal/instance:
uuid: "b2bd236f82de4f98abce9dd4b8639411"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "1ae696bd43e904bcd05e3c9f2a4b0d00"
server_key_iv: "bde7eb80203fea94dd0394c4ea1839fb"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.672075 17710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 16:49:18.675889 17725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:18.677380 17710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:49:18.677559 17710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal
uuid: "b2bd236f82de4f98abce9dd4b8639411"
format_stamp: "Formatted at 2026-09-28 16:49:18 on dist-test-slave-t0tw"
server_key: "1ae696bd43e904bcd05e3c9f2a4b0d00"
server_key_iv: "bde7eb80203fea94dd0394c4ea1839fb"
server_key_version: "encryptionkey@0"
I20260928 16:49:18.677790 17710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:18.713611 17710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:18.718302 17710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:18.718582 17710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:18.736624 17710 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.61:37261
I20260928 16:49:18.736642 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.61:37261 every 8 connection(s)
I20260928 16:49:18.737998 17710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/data/info.pb
I20260928 16:49:18.741758 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:18.745748 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17710
I20260928 16:49:18.745925 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-1/wal/instance
I20260928 16:49:18.746352 27028 external_mini_cluster.cc:1428] Setting key 30ccbc9769c32e96fa7416b50061272a
I20260928 16:49:18.748641 17777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:18.770987 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.751492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.26.101.62:37943)
0928 16:49:18.751972 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:18.751976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:18.753189 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 16:49:18.755187 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:18.755192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:18.755270 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:18.755371 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:18.757443 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.758302 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.760336 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.760839 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.760947 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.765040 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:18.765046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:18.765050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:18.765093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:18.767709 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.768573 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.768578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.768581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.768707 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.769128 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.769132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.769134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.769486 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:18.769627 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:18.770378 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 16:49:18.770613 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.26.101.60@KRBTEST.COM; defaulting to no policy
I20260928 16:49:18.772126 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.752237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37943 (local address 127.0.0.1:57674)
0928 16:49:18.753020 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.753059 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:18.754844 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:18.755653 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:18.755664 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:18.756459 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:18.757199 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.757223 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.758459 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.758464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.760165 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.760185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.761062 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.761068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.761356 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.762120 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:18.762193 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:18.764785 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:18.767863 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.767873 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.767891 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.768449 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.768820 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.768825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.768829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.769017 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.769621 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:18.769629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:18.770016 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 16:49:18.770271 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 16:49:18.770861 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":312,"threads_started":1}
I20260928 16:49:18.774245 17777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.26.101.60@KRBTEST.COM" created.
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:18.788614 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.775017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37286)
0928 16:49:18.775287 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.775303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:18.775393 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:18.776453 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:18.776458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:18.776919 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:18.777057 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.777066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.778434 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.778439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.779374 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.779388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.780178 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.780184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.780367 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.780924 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:18.780943 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:18.783002 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:18.786173 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.786178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.786183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.786530 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.786886 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.786891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.786895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.787041 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.787684 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:18.787688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:18.787829 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 16:49:18.787939 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:49:18.788214 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 16:49:18.788703 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.775341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.26.101.61:37261)
0928 16:49:18.776025 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:18.776031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:18.776054 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:49:18.776101 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:18.776106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:18.776306 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:18.776440 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:18.777222 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.778303 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.779506 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.780044 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.780153 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.783218 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:18.783230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:18.783243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:18.783273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:18.786035 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.786662 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.786669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.786671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.786785 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.787187 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.787192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.787195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.787565 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:18.787751 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:18.788120 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 16:49:18.788354 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":378,"threads_started":1}
I20260928 16:49:18.791213 17777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:18.792335 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.791877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59686)
0928 16:49:18.792041 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.792169 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 16:49:18.792557 17728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:18.794023 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.793698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59700)
0928 16:49:18.793846 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.793921 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:49:18.794231 17777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
Entry for principal kudu/127.26.101.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab.
I20260928 16:49:18.806993 27028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
/tmp/dist-test-taskWH_khZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:34215
--webserver_interface=127.26.101.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.26.101.20:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:49:18.843536 17777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 1
I20260928 16:49:18.844681 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.844217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59708)
0928 16:49:18.844409 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.844519 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 16:49:18.845799 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.845396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59712)
0928 16:49:18.845574 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.845662 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:49:18.846074 17777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:18.914335 17777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 2
I20260928 16:49:18.915550 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.915034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59726)
0928 16:49:18.915293 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.915390 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 16:49:18.916317 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } attempt: 5
I20260928 16:49:18.917268 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.916327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59734)
0928 16:49:18.916510 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.916598 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 16:49:18.917600 17777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Sep 28 16:49:18 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:18.930934 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.917021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.26.101.61:37261)
0928 16:49:18.917512 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:18.917517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:18.917534 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:49:18.917598 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:18.917603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:18.917718 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:18.917817 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:18.918915 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.919658 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.921306 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.921743 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.921856 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.925477 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:18.925482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:18.925486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:18.925521 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:18.928317 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.929026 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.929030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.929032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.929164 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:18.929535 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:18.929538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:18.929540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:18.929841 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:18.929976 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:18.930372 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 16:49:18.930600 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260928 16:49:18.930943 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.916975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37296)
0928 16:49:18.917143 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.917162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:18.917317 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:18.917906 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:18.917913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:18.918473 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:18.918734 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.918749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.919794 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.919800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.921136 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:18.921153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:18.921914 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:18.921921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:18.922148 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:18.922782 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:18.922805 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:18.925269 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:18.928478 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.928485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.928489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.928899 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.929279 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:18.929284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:18.929289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:18.929429 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:18.929963 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:18.929967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:18.930121 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 16:49:18.930243 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 16:49:18.930579 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:49:18.931989 17703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:49:18.932953 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.932563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59744)
0928 16:49:18.932725 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.932813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:49:18.933792 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.933482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59760)
0928 16:49:18.933614 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.933697 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:49:18.933980 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:18.985214 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 1
I20260928 16:49:18.986361 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.985885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59766)
0928 16:49:18.986092 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.986185 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:49:18.987473 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:18.987114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59772)
0928 16:49:18.987254 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:18.987335 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 16:49:18.987684 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:19.008870 17777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 3
I20260928 16:49:19.010195 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.009689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59780)
0928 16:49:19.009924 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.010022 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 16:49:19.011380 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.010885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59786)
0928 16:49:19.011149 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.011242 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 16:49:19.011596 17777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:19.055934 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 2
I20260928 16:49:19.057189 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.056609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59794)
0928 16:49:19.056918 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.057014 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 16:49:19.058210 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.057863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59808)
0928 16:49:19.058016 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.058090 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:49:19.058429 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
W20260928 16:49:19.145812 17785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:49:19.146202 17785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:49:19.146297 17785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:49:19.146415 17785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:49:19.153986 17785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:49:19.154099 17785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:49:19.154191 17785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:49:19.154258 17785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:49:19.154340 17785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:49:19.154830 17777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 4
I20260928 16:49:19.156036 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.155535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59816)
0928 16:49:19.155769 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.155871 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:49:19.156970 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.156645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59820)
0928 16:49:19.156787 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.156860 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:49:19.157163 17777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:19.164644 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 3
I20260928 16:49:19.165277 17785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.101.20:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.26.101.62:37943,127.26.101.61:37261,127.26.101.60:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/krb5kdc/kudu_127.26.101.60.keytab
--principal=kudu/127.26.101.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.101.60:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.101.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 15:46:59 UTC on ccycloud.quasar-awnebh.root.comops.site
build id 4677567
ASAN enabled
I20260928 16:49:19.165726 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.165237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59834)
0928 16:49:19.165423 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.165523 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 16:49:19.166630 17785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:49:19.166719 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.166365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59838)
0928 16:49:19.166543 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.166613 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:49:19.166955 17703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.26.101.60:34215: Network error: Client connection negotiation failed: client connection to 127.26.101.60:34215: connect: Connection refused (error 111)
I20260928 16:49:19.168004 17785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:49:19.179708 17790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:19.179737 17791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:49:19.180387 17793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:49:19.180632 17785 server_base.cc:1034] running on GCE node
I20260928 16:49:19.181483 17785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:49:19.183022 17785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:49:19.184317 17785 hybrid_clock.cc:648] HybridClock initialized: now 1790614159184231 us; error 45 us; skew 500 ppm
I20260928 16:49:19.184669 17785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614159, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:49:19.187968 17785 init.cc:377] Logged in from keytab as kudu/127.26.101.60@KRBTEST.COM (short username kudu)
I20260928 16:49:19.189518 17785 webserver.cc:460] Webserver started at http://127.26.101.60:43273/ using document root <none> and password file <none>
I20260928 16:49:19.190248 17785 fs_manager.cc:359] Metadata directory not provided
I20260928 16:49:19.190343 17785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:49:19.190622 17785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:49:19.193749 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data/instance:
uuid: "88f0822c7b8a4fcbbe566893dc58d270"
format_stamp: "Formatted at 2026-09-28 16:49:19 on dist-test-slave-t0tw"
server_key: "c4fdde0ff4451cc1b487212efc49dd4a"
server_key_iv: "b4bef2c5b3530185c5355476ff923c95"
server_key_version: "encryptionkey@0"
I20260928 16:49:19.194543 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal/instance:
uuid: "88f0822c7b8a4fcbbe566893dc58d270"
format_stamp: "Formatted at 2026-09-28 16:49:19 on dist-test-slave-t0tw"
server_key: "c4fdde0ff4451cc1b487212efc49dd4a"
server_key_iv: "b4bef2c5b3530185c5355476ff923c95"
server_key_version: "encryptionkey@0"
I20260928 16:49:19.199910 17785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 16:49:19.203567 17800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:49:19.205101 17785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 16:49:19.205274 17785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data,/tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal
uuid: "88f0822c7b8a4fcbbe566893dc58d270"
format_stamp: "Formatted at 2026-09-28 16:49:19 on dist-test-slave-t0tw"
server_key: "c4fdde0ff4451cc1b487212efc49dd4a"
server_key_iv: "b4bef2c5b3530185c5355476ff923c95"
server_key_version: "encryptionkey@0"
I20260928 16:49:19.205462 17785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:49:19.228627 17785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:49:19.232802 17785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:49:19.233031 17785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:49:19.250478 17785 rpc_server.cc:307] RPC server started. Bound to: 127.26.101.60:34215
I20260928 16:49:19.250499 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.101.60:34215 every 8 connection(s)
I20260928 16:49:19.251993 17785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/data/info.pb
I20260928 16:49:19.253371 27028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu as pid 17785
I20260928 16:49:19.253541 27028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWH_khZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613789516984-27028-0/minicluster-data/master-2/wal/instance
I20260928 16:49:19.253968 27028 external_mini_cluster.cc:1428] Setting key eed7f425de6f36eb9ead0b04d663f760
I20260928 16:49:19.256372 17852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:49:19.264894 17852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614159, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
I20260928 16:49:19.278007 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.255711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.26.101.60:34215)
0928 16:49:19.256476 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.256491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.256543 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:49:19.258214 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.258238 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.258648 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.259097 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.260953 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.262456 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.264281 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.265093 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.265241 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.270205 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.270213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.270223 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.270262 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.273232 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.273957 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.273964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.273971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.274196 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.274676 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.274680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.274682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.275403 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.275684 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.276115 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.276781 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":337,"threads_started":1}
I20260928 16:49:19.283427 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.268170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.26.101.62:37943)
0928 16:49:19.268402 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.268409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.269017 (+ 608us) server_negotiation.cc:407] Connection header received
0928 16:49:19.269311 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.269317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.269400 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.269479 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.270831 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.271565 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.273021 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.273446 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.273557 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.277167 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.277172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.277176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.277211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.280205 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.281070 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.281076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.281081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.281250 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.281730 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.281736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.281740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.282080 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.282175 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.282858 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.283105 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 16:49:19.283412 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.268055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.62:37943 (local address 127.0.0.1:57686)
0928 16:49:19.268871 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.268925 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:19.269163 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:19.269585 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:19.269592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:19.270341 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:19.270688 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.270698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.271730 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.271734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.272864 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.272877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.273624 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.273631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.273792 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.274421 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:19.274444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:19.276973 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:19.280362 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.280376 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.280383 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.280943 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.281400 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.281405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.281408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.281585 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.282270 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:19.282277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:19.282462 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 16:49:19.282634 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:49:19.282987 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":375,"threads_started":1}
I20260928 16:49:19.285108 17852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.62@KRBTEST.COM
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614159, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:19.296445 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.281433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.26.101.62:37943)
0928 16:49:19.282036 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.282040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.282059 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:49:19.282104 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.282107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.282163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.282256 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.283694 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.284542 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.285860 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.286477 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.286582 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.290862 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.290867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.290871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.290905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.293794 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.294462 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.294468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.294473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.294596 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.294958 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.294962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.294964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.295298 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.295571 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.295830 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.296058 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":275,"threads_started":1}
I20260928 16:49:19.297497 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.285682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.61:37261 (local address 127.0.0.1:37298)
0928 16:49:19.285927 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.285944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:19.286018 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:19.286302 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:19.286305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:19.286633 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:19.286769 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.286786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.287797 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.287801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.288669 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.288682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.289387 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.289393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.289565 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.290061 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:19.290084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:19.292344 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:19.295237 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.295242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.295246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.295605 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.295963 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.295968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.295972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.296117 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.296595 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:19.296600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:19.296742 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 16:49:19.296865 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:49:19.297141 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 16:49:19.297530 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.285840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.26.101.61:37261)
0928 16:49:19.286035 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.286040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.286058 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:49:19.286115 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.286120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.286195 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.286292 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.286918 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.287662 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.288809 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.289244 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.289359 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.292534 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.292539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.292542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.292576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.295111 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.295718 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.295724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.295727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.295859 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.296216 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.296220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.296222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.296477 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.296611 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.296982 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.297181 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:49:19.298426 17852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614159, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.60@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.26.101.61@KRBTEST.COM
I20260928 16:49:19.311610 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.299156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59854)
0928 16:49:19.299382 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.299394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:19.299487 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:19.299749 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:19.299753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:19.300123 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:19.300250 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.300273 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.301236 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.301241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.302106 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.302118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.302784 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.302789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.302953 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.303474 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:19.303490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:19.305467 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:19.309054 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.309058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.309061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.309488 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.309885 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.309891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.309894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.310057 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.310672 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:19.310676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:19.310837 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 16:49:19.310991 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 16:49:19.311281 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 16:49:19.311908 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.299219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.26.101.60:34215)
0928 16:49:19.299342 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.299347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.299453 (+ 106us) server_negotiation.cc:407] Connection header received
0928 16:49:19.299588 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.299592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.299649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.299720 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.300399 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.301120 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.302232 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.302622 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.302725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.305713 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.305719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.305723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.305755 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.308927 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.309603 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.309608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.309611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.309763 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.310173 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.310177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.310179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.310550 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.310693 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.311115 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.311484 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":45}
I20260928 16:49:19.313012 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.298019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.26.101.61:37261)
0928 16:49:19.298226 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.298230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.298247 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:49:19.298331 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.298336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.298396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.298467 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.299859 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.300595 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.301798 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.302221 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.302330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.306508 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.306515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.306520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.306563 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.309601 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.310402 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.310407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.310410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.310574 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.310995 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.310999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.311002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.311361 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.311652 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.312456 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.312690 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 16:49:19.317274 17852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: Bootstrap starting.
I20260928 16:49:19.317530 27028 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:49:19.321126 17852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:19.322175 17852 log.cc:824] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:19.320585 17678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57688
I20260928 16:49:19.324838 17852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: No bootstrap required, opened a new log
I20260928 16:49:19.332762 17852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.333093 17852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f0822c7b8a4fcbbe566893dc58d270, State: Initialized, Role: FOLLOWER
I20260928 16:49:19.333602 17852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.334539 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.334857 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.335883 17852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:49:19.344874 17870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:49:19.345013 17870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:49:19.349573 17703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 4
I20260928 16:49:19.343046 17785 master_runner.cc:186] Error getting master registration for 127.26.101.62:37943: OK, instance_id { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" instance_seqno: 1790614158202662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:19.356581 17785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.62@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
I20260928 16:49:19.366428 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.350768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.26.101.60:34215)
0928 16:49:19.350955 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.350961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.350984 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:49:19.351043 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.351048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.351146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.351246 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.352293 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.353413 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.355288 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.356085 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.356223 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.360185 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.360190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.360193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.360227 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.363664 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.364387 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.364393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.364401 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.364528 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.364892 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.364897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.364900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.365234 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.365404 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.365866 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.366078 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:49:19.366425 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.350242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59864)
0928 16:49:19.350456 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.350478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:19.350609 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:19.351358 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:19.351363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:19.351909 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:19.352109 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.352122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.353570 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.353576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.355056 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.355105 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.356244 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.356250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.356442 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.357217 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:19.357241 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:19.359969 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:19.363808 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.363813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.363818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.364256 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.364634 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.364640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.364644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.364782 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.365471 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:19.365475 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:19.365611 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 16:49:19.365729 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:49:19.366022 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:49:19.371600 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: Bootstrap starting.
I20260928 16:49:19.375065 17703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:19.376228 17703 log.cc:824] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:19.379030 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: No bootstrap required, opened a new log
I20260928 16:49:19.384778 17703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.385121 17703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3a6e62ec944484ab1b8c2c6d87d0d5, State: Initialized, Role: FOLLOWER
I20260928 16:49:19.385613 17703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.386561 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.386978 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.387787 17703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:19.387909 17678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:49:19.388139 17678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:19.391647 17678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.393261 17678 leader_election.cc:288] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [CANDIDATE]: Term 1 election: Requested vote from peers b2bd236f82de4f98abce9dd4b8639411 (127.26.101.61:37261), 88f0822c7b8a4fcbbe566893dc58d270 (127.26.101.60:34215)
I20260928 16:49:19.395093 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "88f0822c7b8a4fcbbe566893dc58d270"
I20260928 16:49:19.395686 17827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:19.400101 17827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3a6e62ec944484ab1b8c2c6d87d0d5 in term 1.
I20260928 16:49:19.401224 17654 leader_election.cc:302] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3a6e62ec944484ab1b8c2c6d87d0d5, 88f0822c7b8a4fcbbe566893dc58d270; no voters:
I20260928 16:49:19.401823 17873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:49:19.407470 17873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3a6e62ec944484ab1b8c2c6d87d0d5, State: Running, Role: LEADER
I20260928 16:49:19.408829 17873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.423013 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3a6e62ec944484ab1b8c2c6d87d0d5. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.423651 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:19.441759 17887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 16:49:19.442884 17657 tablet.cc:2369] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:49:19.455906 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.427164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.26.101.62:37943)
0928 16:49:19.427384 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.427390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.428857 (+ 1467us) server_negotiation.cc:407] Connection header received
0928 16:49:19.429262 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.429268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.429374 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.429455 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.433717 (+ 4262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.434951 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.438759 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.439614 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.439730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.446119 (+ 6389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.446126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.446131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.446174 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.450534 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.452477 (+ 1943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.452482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.452486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.452685 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.453188 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.453194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.453197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.453635 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.453868 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.455319 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.455552 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:49:19.456383 17777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } attempt: 5
I20260928 16:49:19.458688 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.428855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.26.101.61:37261)
0928 16:49:19.429032 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.429038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.434756 (+ 5718us) server_negotiation.cc:407] Connection header received
0928 16:49:19.436228 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.436236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.436331 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.436403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.441607 (+ 5204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.442709 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.445841 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.446472 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.446603 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.450413 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.450419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.450424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.450482 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.454277 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.455828 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.455835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.455838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.456043 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.456510 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.456516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.456519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.457124 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.457397 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.458079 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.458339 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 16:49:19.460223 17887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:49:19.468161 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.435346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.26.101.60:34215)
0928 16:49:19.435554 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.435560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.440951 (+ 5391us) server_negotiation.cc:407] Connection header received
0928 16:49:19.444666 (+ 3715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.444673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.444775 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.444845 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.446800 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.448151 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.450378 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.450997 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.451136 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.454215 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.454222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.454227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.454271 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.460820 (+ 6549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.464758 (+ 3938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.464783 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.464787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.464995 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.465755 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.465761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.465764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.466180 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.466446 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.467453 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.467736 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:49:19.485118 17806 tablet.cc:2369] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 16:49:19 dist-test-slave-t0tw krb5kdc[17623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790614158, etypes {rep=17 tkt=17 ses=17}, kudu/127.26.101.61@KRBTEST.COM for kudu/127.26.101.60@KRBTEST.COM
I20260928 16:49:19.480321 17636 master_runner.cc:186] Error getting master registration for 127.26.101.61:37261: OK, instance_id { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" instance_seqno: 1790614158731610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:49:19.491876 17636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:49:19.492719 17827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [term 1 FOLLOWER]: Refusing update from remote peer 7c3a6e62ec944484ab1b8c2c6d87d0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:19.493976 17879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:19.507953 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.456965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.101.60:34215 (local address 127.0.0.1:59870)
0928 16:49:19.457868 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 16:49:19.457887 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:49:19.457987 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:49:19.471255 (+ 13268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:49:19.471262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:49:19.471809 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:49:19.472054 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.472069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.473870 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.473876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.481162 (+ 7286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:49:19.481184 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.482509 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.482516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:49:19.482755 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.485666 (+ 2911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:49:19.485692 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:49:19.489011 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:49:19.500584 (+ 11573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.500590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.500594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.501201 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.504987 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:49:19.504993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:49:19.504997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:49:19.505219 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:49:19.506713 (+ 1494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:49:19.506718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:49:19.506946 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 16:49:19.507141 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:49:19.507541 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772}
I20260928 16:49:19.510628 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:19.458320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.26.101.60:34215)
0928 16:49:19.463593 (+ 5273us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.463600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.463624 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:49:19.463682 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.463687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.463774 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.472226 (+ 8452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.472494 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.473711 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.481389 (+ 7678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:19.482083 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:19.482205 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:19.489262 (+ 7057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:49:19.489270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:49:19.489275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:49:19.489388 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:49:19.500357 (+ 10969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.501520 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.501529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.501533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.504753 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:49:19.505465 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:49:19.505472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:49:19.505476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:49:19.506450 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:49:19.508952 (+ 2502us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:19.509228 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 16:49:19.509759 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5132,"thread_start_us":4925,"threads_started":1}
I20260928 16:49:19.518205 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: Bootstrap starting.
I20260928 16:49:19.525239 17777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: Neither blocks nor log segments found. Creating new log.
I20260928 16:49:19.527587 17777 log.cc:824] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: Log is configured to *not* fsync() on all Append() calls
I20260928 16:49:19.531149 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3a6e62ec944484ab1b8c2c6d87d0d5. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.531549 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.547597 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.547967 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:49:19.548702 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.548978 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.549839 17887 catalog_manager.cc:1347] Generated new cluster ID: e2539ec543e34dc1987cf644d5028373
I20260928 16:49:19.549947 17887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:49:19.553023 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: No bootstrap required, opened a new log
I20260928 16:49:19.574113 17887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:49:19.576129 17887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:49:19.587136 17777 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.588287 17777 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2bd236f82de4f98abce9dd4b8639411, State: Initialized, Role: FOLLOWER
I20260928 16:49:19.589457 17777 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } }
I20260928 16:49:19.591831 17887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5: Generated new TSK 0
I20260928 16:49:19.593034 17887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:49:19.595156 17903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.595707 17903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.597087 17777 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:49:19.597186 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2bd236f82de4f98abce9dd4b8639411"
I20260928 16:49:19.597699 17752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:49:19.601363 17752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3a6e62ec944484ab1b8c2c6d87d0d5 in term 1.
I20260928 16:49:19.603230 17751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [term 1 FOLLOWER]: Refusing update from remote peer 7c3a6e62ec944484ab1b8c2c6d87d0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:49:19.606803 17873 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:49:19.637337 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.637856 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.638219 17903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3a6e62ec944484ab1b8c2c6d87d0d5. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:19.638497 17903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:49:19.709700 17922 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 16:49:19.727643 17857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.724752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.26.101.62:37943)
0928 16:49:19.724960 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.724967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.725525 (+ 558us) server_negotiation.cc:407] Connection header received
0928 16:49:19.725822 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.725828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.725949 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.726027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.727322 (+ 1295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57700: BlockingRecv error: recv got EOF from 127.0.0.1:57700 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 16:49:19.730448 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.725573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.26.101.61:37261)
0928 16:49:19.725768 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.725774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.728670 (+ 2896us) server_negotiation.cc:407] Connection header received
0928 16:49:19.728951 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.728958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.729055 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.729129 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.730184 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37328: BlockingRecv error: recv got EOF from 127.0.0.1:37328 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 16:49:19.733785 17895 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.731759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.26.101.60:34215)
0928 16:49:19.731953 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.731958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.732229 (+ 271us) server_negotiation.cc:407] Connection header received
0928 16:49:19.732382 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.732387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.732495 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.732597 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.733548 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59886: BlockingRecv error: recv got EOF from 127.0.0.1:59886 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:49:19.769508 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.762706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.26.101.61:37261)
0928 16:49:19.762940 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.762947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.767824 (+ 4877us) server_negotiation.cc:407] Connection header received
0928 16:49:19.768079 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.768085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.768209 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.768329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.769245 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37338: BlockingRecv error: recv got EOF from 127.0.0.1:37338 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260928 16:49:19.776890 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.774641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.26.101.61:37261)
0928 16:49:19.774819 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.774826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.775026 (+ 200us) server_negotiation.cc:407] Connection header received
0928 16:49:19.775627 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.775634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.775729 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.775841 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.776661 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37344: BlockingRecv error: recv got EOF from 127.0.0.1:37344 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 16:49:19.785792 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.783238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.26.101.61:37261)
0928 16:49:19.783461 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.783468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.784090 (+ 622us) server_negotiation.cc:407] Connection header received
0928 16:49:19.784397 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.784403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.784517 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.784633 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.785519 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 16:49:19.796183 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.794150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.26.101.61:37261)
0928 16:49:19.794344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.794350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.794526 (+ 176us) server_negotiation.cc:407] Connection header received
0928 16:49:19.794850 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.794856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.794977 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.795130 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.795952 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37364: BlockingRecv error: recv got EOF from 127.0.0.1:37364 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 16:49:19.811725 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.809326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.26.101.61:37261)
0928 16:49:19.809542 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.809548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.809960 (+ 412us) server_negotiation.cc:407] Connection header received
0928 16:49:19.810363 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.810369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.810470 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.810583 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.811474 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37376: BlockingRecv error: recv got EOF from 127.0.0.1:37376 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:49:19.834311 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.831848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.26.101.61:37261)
0928 16:49:19.832063 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.832070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.832657 (+ 587us) server_negotiation.cc:407] Connection header received
0928 16:49:19.832881 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.832888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.832992 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.833109 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.834065 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 16:49:19.873579 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.871060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.26.101.61:37261)
0928 16:49:19.871298 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.871306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.871884 (+ 578us) server_negotiation.cc:407] Connection header received
0928 16:49:19.872107 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.872113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.872240 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.872361 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.873298 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37390: BlockingRecv error: recv got EOF from 127.0.0.1:37390 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 16:49:19.943101 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:19.941206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.26.101.61:37261)
0928 16:49:19.941416 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:19.941422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:19.941718 (+ 296us) server_negotiation.cc:407] Connection header received
0928 16:49:19.941894 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:19.941900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:19.941993 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:19.942103 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:19.942842 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37404: BlockingRecv error: recv got EOF from 127.0.0.1:37404 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 16:49:20.076489 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:20.074432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.26.101.61:37261)
0928 16:49:20.074661 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.074668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.075069 (+ 401us) server_negotiation.cc:407] Connection header received
0928 16:49:20.075304 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.075308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.075383 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.075472 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:20.076282 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37406: BlockingRecv error: recv got EOF from 127.0.0.1:37406 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 16:49:20.080695 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3a6e62ec944484ab1b8c2c6d87d0d5" member_type: VOTER last_known_addr { host: "127.26.101.62" port: 37943 } } peers { permanent_uuid: "b2bd236f82de4f98abce9dd4b8639411" member_type: VOTER last_known_addr { host: "127.26.101.61" port: 37261 } } peers { permanent_uuid: "88f0822c7b8a4fcbbe566893dc58d270" member_type: VOTER last_known_addr { host: "127.26.101.60" port: 34215 } } }
I20260928 16:49:20.081028 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3a6e62ec944484ab1b8c2c6d87d0d5 [sys.catalog]: This master's current role is: LEADER
W20260928 16:49:20.338390 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:49:20.336474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.26.101.61:37261)
0928 16:49:20.336741 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.336748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.336958 (+ 210us) server_negotiation.cc:407] Connection header received
0928 16:49:20.337145 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.337150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.337226 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.337307 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:49:20.338149 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37412: BlockingRecv error: recv got EOF from 127.0.0.1:37412 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 16:49:20.346980 17870 catalog_manager.cc:1259] Loaded cluster ID: e2539ec543e34dc1987cf644d5028373
I20260928 16:49:20.347154 17870 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: loading cluster ID for follower catalog manager: success
I20260928 16:49:20.349680 17870 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: acquiring CA information for follower catalog manager: success
I20260928 16:49:20.351573 17870 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88f0822c7b8a4fcbbe566893dc58d270: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:20.714212 17922 catalog_manager.cc:1259] Loaded cluster ID: e2539ec543e34dc1987cf644d5028373
I20260928 16:49:20.714368 17922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: loading cluster ID for follower catalog manager: success
I20260928 16:49:20.717200 17922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: acquiring CA information for follower catalog manager: success
I20260928 16:49:20.719161 17922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2bd236f82de4f98abce9dd4b8639411: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:49:20.870308 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:20.864017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.26.101.61:37261)
0928 16:49:20.864812 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.864816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.864833 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:49:20.864895 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.864898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.865085 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.865202 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:20.865632 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.866500 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.867913 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.868349 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.868461 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:20.869010 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:20.869625 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:20.869795 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:20.869887 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 16:49:20.869976 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":484,"threads_started":1}
I20260928 16:49:20.883432 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:20.876379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.26.101.60:34215)
0928 16:49:20.876901 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.876907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.877262 (+ 355us) server_negotiation.cc:407] Connection header received
0928 16:49:20.877481 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.877485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.877649 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.877855 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:20.878310 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.879109 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.881049 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.881459 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.881552 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:20.882154 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:20.882719 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:20.882842 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:20.882992 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:49:20.883069 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260928 16:49:20.898496 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:20.890858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.26.101.61:37261)
0928 16:49:20.891051 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.891057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.891741 (+ 684us) server_negotiation.cc:407] Connection header received
0928 16:49:20.892007 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.892013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.892100 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.892161 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:20.892871 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.894037 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.896199 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.896865 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.896962 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:20.897713 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:20.897850 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:20.897909 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:20.898111 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:49:20.898201 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:49:20.903290 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:20.891903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.26.101.60:34215)
0928 16:49:20.895922 (+ 4019us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.895928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.895949 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:49:20.896263 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.896269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.896382 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.896471 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:20.897948 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.899136 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.901372 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.901963 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.902098 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:20.902544 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:20.902664 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:20.902738 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:20.902884 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 16:49:20.902968 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3872}
I20260928 16:49:20.904110 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:49:20.890144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.26.101.62:37943)
0928 16:49:20.896567 (+ 6423us) server_negotiation.cc:206] Beginning negotiation
0928 16:49:20.896573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:49:20.896594 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:49:20.896706 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:49:20.896711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:49:20.896931 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:49:20.897096 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:49:20.897948 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.899045 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.901390 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:49:20.901794 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:49:20.901903 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:49:20.902498 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:49:20.902913 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:49:20.903675 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0928 16:49:20.903730 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 16:49:20.903826 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6287,"thread_start_us":6417,"threads_started":1}
I20260928 16:49:20.908771 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17636
I20260928 16:49:20.909891 17698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:21.043640 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17636
I20260928 16:49:21.063575 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17710
I20260928 16:49:21.064884 17772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:21.181087 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17710
I20260928 16:49:21.199399 27028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu pid 17785
I20260928 16:49:21.200562 17847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 16:49:21.307153 27028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWH_khZ/build/asan/bin/kudu with pid 17785
2026-09-28T16:49:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4880 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9472 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (370996 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST